AWS Data Engineering Project: End-to-End RDS to S3 with Glue

Unlock Real-World Data Engineering Skills with Our Hands-On AWS Project

Transform into a skilled data engineer by building a comprehensive and realistic AWS data engineering project that ingests healthcare data from Amazon RDS PostgreSQL into an Amazon S3 data lake using AWS Glue.

Join this project-based course and work through the project as a data engineer, from understanding the business context and source systems to defining the pipeline contract, making design decisions, setting up AWS, implementing the solution, validating the output, and scheduling repeatable runs.

What You'll Achieve in This Project:

• Prepare the Source: Set up a PostgreSQL database in Amazon RDS and get familiar with the data structure.

• Build the Glue Job: Ingest multiple source tables using AWS Glue and PySpark, and learn how to handle complex data transformations.

• Load Incrementally: Use watermarks to process new and updated records, ensuring that your pipeline is efficient and scalable.

• Store the Data: Write the output to Amazon S3 as Parquet files, optimizing your data storage for future analysis.

• Track Each Run: Add ingestion metadata for auditing and validation, giving you complete visibility into your pipeline's performance.

• Catalog the Data: Use AWS Glue Crawlers and the Glue Data Catalog to discover and understand your data, making it easier to analyze and visualize.

• Validate the Output: Query the ingested data using Amazon Athena, ensuring that your pipeline is producing accurate and reliable results.

• Schedule the Pipeline: Use Glue Workflows and Triggers for repeatable runs, automating your pipeline and reducing manual effort.

More Than Just Isolated Service Demonstrations:

You'll build the project inside your own AWS account using the provided source code, healthcare sample data, configuration files, and infrastructure setup scripts. This hands-on approach will give you a deep understanding of how each AWS service works together to deliver a comprehensive data engineering solution.

The focus is not only on getting the pipeline to run but also on understanding the business context, source systems, and design decisions that drive the pipeline's architecture.

By the end of this course, you'll have a complete AWS data engineering project that you can practice, adapt for your portfolio, and explain clearly during interviews.

This course contains a promotion and is a limited-time offer.