Build a production-style real-time streaming data pipeline on Azure, right from Data Ingestion to live dashboards. In this hands-on project, we ingest Live Weather API events through two paths (Databricks PySpark and Azure Functions), publish to Azure Event Hubs, process with Microsoft Fabric Eventstream, land data in Eventhouse (Kusto), and visualize insights in Power BI with real-time alerts via Data Activator.
You’ll learn how to provision the environment correctly (Resource Groups, Databricks workspace/cluster, Event Hubs namespace & hub, Key Vault, Fabric workspace), manage secrets securely, and wire services together for a reliable streaming architecture. We’ll write KQL for fast analysis, and apply cost-savvy choices for clusters, Throughput Units, and storage.
By the end, you will be able to:
- Design a secure, scalable streaming architecture on Azure.
- Implement ingestion with Databricks (PySpark) and Azure Functions.
- Stream events into Event Hubs and route them with Fabric Eventstream.
- Load/query data in Eventhouse using KQL.
- Build live Power BI dashboards and trigger alerts with Data Activator.
- Monitor, troubleshoot, and optimize for cost and performance.
- Perform final end-to-end pipeline testing, from data ingestion all the way to the report updating with the latest data.
This project is perfect for anyone who wants a clear, practical understanding of how streaming ETL pipelines are built in real-world projects.
