Udemy – Databricks Data Engineering: The Real World Simulator

Helios Trading Corporation needs a Data Engineer. That is you. Helios ships parts across six depots in the solar system. Ten source feeds land every day. Nobody has built the pipeline yet. This is not a tutorial you follow. It is a job you do.

Simulate, then Execute

Every build starts as a conversation. A short AI Role Play with a colleague, and goals you are measured against. Then you build what you talked through. Most labs are work tickets: context, task, and acceptance criteria that run as real checks in the notebook.

What you will build:

• 1. Design Meet Helios, walk the ten source feeds with the person who owns them, and agree the Medallion lakehouse design that every later section implements.
• 2. Databricks Jobs The classic build. Auto Loader into Bronze, Structured Streaming and MERGE into Silver, SCD Type 2 for prices and customer tiers, a Gold star schema with point in time pricing, then a five task orchestrated Job with retries, alerts and a schedule.
• 3. Lakeflow Spark Declarative Pipelines The same platform, rebuilt declaratively. Streaming tables, expectations as quality gates, AUTO CDC. Then an inherited pipeline fails on a night run and you diagnose it from the event log.
• 4. Genie and AI/BI Dashboards A semantic layer of Metric Views over Gold, so one definition of revenue feeds both a Genie Space and a dashboard. Then you make the case for self serve to the Commercial Director.
• 5. Lakebase and a Databricks App Sync curated Gold into Lakebase and deploy a live Depot Operations Console.