Welcome to Big Data Analytics & Designing ETL Pipelines with Pyspark course. This is a comprehensive project based course where you will learn how to process and analyze large datasets, perform complex data transformation, and build ETL pipelines. This course is a perfect combination between big data and Pyspark, making it an ideal opportunity to practice your programming skills while improving your technical knowledge in data engineering. In the introduction session, you will learn the basic fundamentals of big data analytics and ETL, you will learn big data workflow and the difference between ETL and ELT.
Then, in the next section, we will learn basic concepts of Pyspark like creating Pyspark DataFrame, reading CSV files, DataFrame operations, selecting columns, filtering, sorting data, converting data types, handling missing values and removing duplicates. Afterward, in the next section, we will learn how to transform data using aggregation techniques and combine datasets using union and join operations. These skills will enable us to efficiently process large datasets and prepare data for developing ETL pipelines.
Following that, we are going to design a complete ETL pipeline for flight and airport data. We will start with data extraction and ingestion, where we will get data from three different sources, flight records CSV, airport information CSV, and aircraft specifications JSON.
