Build a complete real-time data engineering pipeline using Python, DuckDB, dbt, Streamlit, and live API data. Data engineering is about much more than writing SQL or moving data from one place to another. Modern data engineers need to understand how data is collected, stored, transformed, tested, automated, and delivered to users. In this hands-on course, you will build a practical end-to-end data engineering project from scratch. We will use a live cryptocurrency market API as our data source, allowing you to work with continuously changing real-world data rather than static CSV files. You will start by setting up your Python development environment and installing the tools required for the project.
Next, you will build a Python ingestion process that connects to a REST API, retrieves live market data, processes the JSON response with Pandas, and incrementally loads each new snapshot into DuckDB. You will then implement a modern Bronze, Silver, and Gold architecture. The Bronze Layer will preserve historical API data. Using dbt, you will transform this raw data into a clean and standardized Silver Layer before creating an analytics-ready Gold Layer containing the latest market information. You will also learn how to define dbt sources, create model dependencies, use SQL window functions, and implement automated data quality tests. But we won’t stop at transformation.
