Hi guys ! It’s me again . As scheduled, today we will go together in a series on data warehosue. In the previous sections, we learned about upserting data into the Data warehouse. In this section, we will learn about the modern architecture of the data warehouse.
Note :Those who have not read the previous section can review follow this link:
Big Data - Data Warehouse - Upsert Table In Data Warehouse(Part 2)
A picture of more than a thousand words
Now we will go through each step
- Combine all our data type((logs, files, and media…) from manny source using Azure Data Factory into Azure Blob Storage.
From Azure Blob Storage, we see two ways out. The first direction is pouring into the Azure Databricks and the second direction is pouring directly into Azure Synapse Analytic.
OK, we will see the first direction
- Leverage data in Azure Blob Storage( or Azure Data Lake Storage) to perform scalable analytics with Azure Databricks and achieve cleansed and transformed data.
Note : Azure Databricks contains the technologies and capabilities of the complete open source Apache Spark cluster. So Azure Databricks can use distributed machine learning algorithms to give us insight into the data.
- . Here you see data from Azure Databricks into PowerBI, which is because above we already know that Azure Databricks has the full capabilities of a Spark cluster so we can run queries directly or give visually insight about data.
This is the simplest flow in this architecture 1 --> 2 --> 5
Let’s continue !
A second and third flow from Azure Data Lake Storage or Azure Databricks pour into Azure Synapse Analytics.
Let’s see which flow from Azure Blob straight to Azure Synapse Analytics !
Azure Synapse Analytics combine data from Azure Blob with existing structured data, creating one hub for all our data.
We can say that Azure Synapse is Azure SQL Data Warehouse
And the data from the data warehouse is now ready for analysis :))
Azure Analysis Services provides enterprise-grade data models in the cloud. The data model provides an easier and faster way for users to perform ad hoc data analysis using tools like Power BI and Excel.
The final flow is similar to the one above, except that the data can be cleaned up and transformed in Azure Databricks.
OK, So that’s all for this section !
In the next section, we will learn about Azure Data Factory, a very important component in this architecture
See you soon !