This project implements an end-to-end event-driven data pipeline using AWS services and Apache Spark.
Architecture
S3 Bronze Layer for raw data
Lambda for orchestration
AWS Glue (Spark) for transformations
S3 Silver Layer in Parquet format
Future extension to Postgres and Power BI
Event Flow
CSV file uploaded to S3 Bronze bucket
S3 triggers Lambda
Lambda triggers AWS Glue job
Glue reads CSV, applies schema and transformations
Output written to S3 Silver as Parquet
Tech Stack
AWS S3
AWS Lambda
AWS Glue (Apache Spark)
IAM
Python
PySpark
Key Design Decisions
Parquet used for efficient analytics
Glue used instead of Lambda for large-scale transformations
Event-driven architecture (no cron jobs)
u/Master_Ad2559 — 9 days ago