Udemy – AWS Data Engineering Project: End-to-End RDS to S3 with Glue

Build a realistic AWS data engineering project that ingests healthcare data from Amazon RDS PostgreSQL into an Amazon S3 data lake using AWS Glue. You will work through the project as a data engineer—from understanding the business context and source systems to defining the pipeline contract, making design decisions, setting up AWS, implementing the solution, validating the output, and scheduling repeatable runs.

In this project, you will:

• Prepare the source: Set up a PostgreSQL database in Amazon RDS
• Build the Glue job: Ingest multiple source tables using AWS Glue and PySpark
• Load incrementally: Use watermarks to process new and updated records
• Store the data: Write the output to Amazon S3 as Parquet files
• Track each run: Add ingestion metadata for auditing and validation
• Catalog the data: Use AWS Glue Crawlers and the Glue Data Catalog
• Validate the output: Query the ingested data using Amazon Athena
• Schedule the pipeline: Use Glue Workflows and Triggers for repeatable runs