Chapter 4: Building ETL Pipelines With AWS Glue
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Aws glue
The Data Consumption Layer - Querying With Trino
You’ve built your ingestion, you’ve processed your data with Spark, and it’s all sitting neatly in your S3 “Gold” bucket. Now what? You can’t ask every business analyst to learn PySpark just to see last month’s sales.
Data Engineering With AWS Chapter 7 Part 2: Transforming Data - Optimization and Business Logic
This is post 12 in my Data Engineering with AWS retelling series.
In Part 1, we covered the generic data preparation transforms: converting to Parquet, partitioning, PII protection, and data cleansing. Those transforms work on individual datasets and do not need much business context. Now we get to the transforms that actually create business value. The ones that combine multiple datasets, add context, flatten structures, and produce the tables that analysts and dashboards consume.
Data Engineering With AWS Chapter 3 Part 2: The AWS Toolkit - Analytics and Processing
In Part 1 we covered how data gets into AWS. Now comes the good part: what do you actually do with it once it is there? This post covers the services for transforming raw data, orchestrating multi-step pipelines, and letting people query and visualize the results.
Elastic MapReduce: Running Hadoop in the AWS Cloud
Previous: Mastering AWS for Big Data: EC2, S3, and EMR
In the last post, we looked at the basic building blocks of AWS: EC2 and S3. But if you’re trying to run a massive Hadoop or Spark cluster, you don’t really want to be manually installing software on hundreds of individual EC2 instances. That’s where Amazon EMR (Elastic MapReduce) comes in.