“Without big data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway.”
— Geoffrey Moore
Becoming a data-driven enterprise is certainly a winning strategy, while a data warehouse is the driving power for informed strategic decisions based on predictive analytics and meaningful reporting solutions. Should we keep data in-house or outsource it to a hosting provider? Which infrastructures are viable for our business: on-premises or cloud-based? How to build a data warehouse from scratch? These are the major questions customers address to Oxagile’s data engineers.
We highlighted customers’ concerns about data warehouses (DWH) during a Q&A session with Alena Gabaraeva, Data Engineer at Oxagile. She shed light on what makes up a data warehouse architecture and how a typical data warehouse implementation plan may look.
Why build a data warehouse for businesses?
Data warehousing is beneficial for companies of all sizes. Some of the advantages include the following:
- Informed decision making. Ready-made access to corporate data simplifies the generation of ad hoc reports.
- Accurate data management. It helps collect all types of data from multiple sources onto a single database that can be used to retrieve data for analysis.
- Saving time on processing data from multiple sources. Storing all the required data in one place reduces time on retrieving data and the total turnaround time for analytics and reporting.
Further reading: Data analytics use cases
Learn how data can be transformed into business opportunities.
Data warehouse architecture, or what components will you need?
Expert comment:
“A data warehouse architecture includes technological elements that can be adjusted to the specific needs of an organization. Still, the core of the architecture is made up of the components below:
- Source data component serving to gather data into a warehouse from multiple operational systems and external sources.
- Data staging component that converts the data into the format relevant for querying and analysis.
- Data storage component that handles the DWH data.
- Information delivery component enables the process of subscribing for the data warehouse files in accordance with a custom algorithm.
- Metadata component for keeping all the data about logical structures, logs, addresses, etc.
- Data marts are the subsets of a data warehouse used for the reports for different units.
- Management and control component that coordinates all the functions within the big data storage system.”
What are the approaches to building a data warehouse?
There are two traditional design methodologies used to build a data warehouse architecture: Inmon’s approach and Kimball’s approach. When applying the former, data architects design a centralized storage first and then create data marts from the summarized data warehouse and metadata. The latter follows the opposite process: initial creating of data marts and then the development of a data warehouse database from independent data marts.
Expert comment:
“Both methods work for business, but the market instability makes businesses adapt by being agile. That’s why a new methodology, Data Vault modeling, is gaining popularity due to its agility principles.
The methodology ensures the DWH flexibility and scalability through opportunities for refactoring and familiar architectures for further use cases within a company. Finally, the main advantage of this approach is a faster time-to-market, which may be critical for specific business domains: those who can respond to changes instantly stay competitive. That’s a fact.”
Data warehouse implementation plan with the expert’s hands-on flashbacks
Every business case is all about specific needs, so you can’t just apply the same steps to each project. While talking to our expert, we dived into the flashbacks from the project she was deeply involved in to clearly understand each step of a data warehouse implementation plan.
Let us briefly tell you the context of the story. It’s quite a common case for businesses storing data in multiple sources. To subscribe the existing reporting to a centralized data warehouse system, such companies have to integrate all the data into the new DWH.
How does it typically work? A common warehouse implementation project includes such steps as:
- Budgeting
- Data sources analysis
- Data warehouse architecture design
- Development and implementation
- Launching a data warehouse
Step 1. On-premises, or cloud deployment, that is the question
The very first factor to decide on is the availability of data warehouse deployment options. That’s why be sure you explore the data storage restrictions within the local jurisdiction. Let’s say your business is in the USA; in this case, you must consult on the laws at the state level as there is no single principal data protection legislation. It may occur that you would be limited to keeping data servers located physically within the state (country) of your jurisdiction, so on-premises deployment will be the only possible option.
What if you’re given a free hand to choose among cloud-based and on-premises infrastructures? It’s worth analyzing both options’ pros and cons related to finance and maintenance. For instance, cloud server providers commit to maintaining all hardware and fixing data processing issues, but it may take time. For some organizations, the inability to tackle the issue on site as soon as possible might be critical. However, hosting corporate servers requires an in-house IT department to tackle all the issues on their own, which may be costly enough for small and medium-sized businesses.
The expert’s project flashback:
“When working on the DWH implementation for an eCommerce company, we analyzed their business specifics and country laws related to data storage and processing. Since the company’s target clients and location were within the same jurisdiction, that didn’t allow keeping data logs on servers located outside that country, so we had to establish an on-premises infrastructure.”
Need an expert opinion?
Do you feel unsure you can define which infrastructure, on-premises or cloud, is optimal for your business? Let us know! Oxagile’s experts will be happy to research this for you.
Step 2: What does your data look like? Measure data use and understand its complexity
Hardware-related things are done; it’s time to choose an appropriate software stack. Data analysis is a must-have preparation step for the data warehouse implementation project. To correlate the vendor’s DWH software with the potential costs, you need to analyze the answers to the questions, “What’s the amount of data aggregated by the business, and what’s the growth potential?”, “What’s the data that needs processing, and where does it come from?”
- Food for thought #1: Data aggregation growth rates. Why is it critical? That’s quite simple. The more data is aggregated, the larger data storage capacities are required. You should think pragmatically and consider opportunities for smooth and cost-effective scaling.
- Food for thought #2: Complexity of data sources. Why does it make sense to start this data analysis before data warehouse implementation? It’s about data formats. The data warehouse can be fueled by several data sources using different types of files, which defines the complexity of integration with all the data sources.
The expert’s project flashback:
“As I’ve mentioned above, we couldn’t deploy the data warehouse in the cloud, so after analyzing client data, we chose Oracle and Cloudera. Still, the final determinant was cost-effectiveness, which brought a victory to Cloudera. It allowed us not only to save costs but also ensure the variety of data integrations and seamless growth of the DWH infrastructure, making it easier to implement a data warehouse.”
Step 3: Data mapping enters the scene to design the data warehouse architecture
A data warehouse architecture serves the foundation for its implementation. In addition to an in-depth analysis of data sources, data architects cover the processes below to design the data warehouse architecture:
- Map data objects into the data warehouse
- Design data models for the data warehouse
- Define the dependencies of data entities
- Design ETL/ELT processes to manage data integration and data flows
The expert’s project flashback:
“At this stage, data analysts and data architects worked in tandem: the former analyzed the sources and mapped data objects, while the latter designed them for the data staging and other components. By designing the architecture, they provided the data warehouse with high-quality aggregated data and identified source-to-target data flow scenarios.”
Step 4: Ready, steady, go: develop, implement, and launch
Finally, it’s time for production. The key processes covered at this stage include:
- Configuring the DWH platform
- Integrating with the existing data infrastructure
- Assessing the quality of the migrated data
- Building ETL/ELT pipelines and ETL/ELT testing
- Integrating with BI solutions such as Looker or Tableau
- Testing the DWH performance
- Ensuring all the stakeholders have access to the DWH
- Conducting user training sessions
Use case in point: Big data-based online video monitoring solution
Get inspired by another piece of content on how your company’s departments can leverage the data carefully stored in your DWH.
Final takeaways
So, is the data warehouse the future for data-driven businesses?
Expert opinion
“Though data warehousing might be costly, it’s a competitive advantage for businesses in the long term. The amount of data is tremendously growing, and those who don’t analyze it to get insights are likely to lag. Inspired by an increasing trend of maximum data collection, companies analyze terabytes of data. When you own the data and dive deep into understanding it, you know what would await you in the future, but you should understand that without a single data warehouse, it’s hard to get complex insights.”
Are you looking for professional data engineering services? Contact Oxagile to discuss your current needs.
