Home » Tags » Aws

Aws

Airflow on AWS: Fargate, S3, Glue, and a Movie Ranking DAG

Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901

Chapter 16 is where the cloud chapters get concrete. Harenslak and deRuiter map every Airflow component to an AWS service, then build a working movie-ranking pipeline with S3, Glue, and Athena.

Data Engineering With AWS: Closing Thoughts on This Book Retelling Series

We made it. Twenty posts. Fourteen chapters. One very thorough book about building data pipelines on Amazon Web Services.

When I started this retelling series back in December, I said the goal was to walk you through “Data Engineering with AWS” by Gareth Eagar in plain language, one chapter at a time. No jargon walls. No assumed knowledge. Just the concepts, the tools, and how they fit together. I hope I delivered on that.

Data Engineering With AWS Chapter 14: Wrapping Up the Learning Journey

This is post 20 in my Data Engineering with AWS retelling series.

We made it. Thirteen chapters of pipelines, transforms, orchestration, security, querying, visualization, and machine learning. Chapter 14 is the final chapter. It does not introduce a new AWS service. Instead, it zooms out. Way out. It looks at how data engineering works in the real world, shows you case studies from Spotify and Netflix, and points you toward where the field is heading next.

Data Engineering With AWS Chapter 12: Visualizing Data With Amazon QuickSight

This is post 18 in my Data Engineering with AWS retelling series.

We have spent eleven chapters ingesting data, transforming data, cataloging data, querying data. But here is a simple truth: nobody wants to stare at 10,000 rows in a spreadsheet. Our brains are not built for that. We process pictures way faster than text. A well-designed chart can tell you in two seconds what would take twenty minutes to figure out from raw numbers.

Data Engineering With AWS Chapter 9 Part 2: Bridging Data Lake and Data Warehouse

This is post 15 in my Data Engineering with AWS retelling series.

In Part 1, we looked at Redshift internals – clusters, slices, distribution styles, sort keys. All the pieces that make a data warehouse fast. But a warehouse sitting in isolation is not very useful. Data needs to flow in from your data lake, and sometimes it needs to flow back out. Part 2 of Chapter 9 covers that bridge between S3 and Redshift, including Redshift Spectrum, the COPY and UNLOAD commands, and a hands-on exercise that ties it all together.

Scaling to the Cloud With Amazon EKS

Testing things locally with Kind is great, but big data usually needs big iron. In this part of the hands-on journey, Neylson Crepalde shows us how to scale up to a managed cloud environment.

Data Engineering With AWS Chapter 7 Part 2: Transforming Data - Optimization and Business Logic

This is post 12 in my Data Engineering with AWS retelling series.

In Part 1, we covered the generic data preparation transforms: converting to Parquet, partitioning, PII protection, and data cleansing. Those transforms work on individual datasets and do not need much business context. Now we get to the transforms that actually create business value. The ones that combine multiple datasets, add context, flatten structures, and produce the tables that analysts and dashboards consume.

Data Engineering With AWS Chapter 6 Part 1: Ingesting Batch Data

This is post 9 in my Data Engineering with AWS retelling series.

You have your whiteboard architecture from Chapter 5. You know who your data consumers are and what they need. Now it is time to actually move data. Chapter 6 covers data ingestion – getting data from wherever it lives into your AWS data lake. This first part focuses on batch ingestion from databases and files. Part 2 covers streaming.

Data Engineering With AWS Chapter 1: What Even Is Data Engineering?

If someone told you twenty years ago that data would become more valuable than oil, you would have laughed. But here we are. The most valuable companies on the planet are not drilling for crude. They are collecting, processing, and squeezing insights out of massive piles of data. And behind every one of those companies, there is a team of data engineers making it all work.

About

About BookGrill.net

BookGrill.net is a technology book review site for developers, engineers, and anyone who builds things with code. We cover books on software engineering, AI and machine learning, cybersecurity, systems design, and the culture of technology.

Know More

Category

Tags View all tags

Book-Retelling Data-Engineering Python Aws Apache-Airflow Chisom-Nwokwu Data-Science Data-Engineering-With-Python Paul-Crickard Bas-Harenslak Data-Pipelines-With-Apache-Airflow Julian-Deruiter Big-Data-on-Kubernetes Neylson-Crepalde Nlp Big-Data Algorithms Haystack Laura-Funderburk Llm Nlp-Llm-Pipelines Sam-Morley Adi-Wijaya Cloud-Computing Data-Engineering-Gcp Data-Structures Google-Cloud-Platform Bhagvan-Kommadi Go-Programming Golang Learn-Golang Data-Engineering-Aws Gareth-Eagar Data-Engineering-for-Beginners-Chisom-Nwokwu Data-Pipelines C-Plusplus-Programmers-Mindset Daniel-Vaughan Data-Science-Foundations Data-Science-the-Hard-Parts Ian-Huke Stephen-Mariadas Data-Engineering-for-Beginners Data-Analytics-for-Finance-Using-Python Devops Infrastructure-as-Code Nitin-Jaglal-Untwal Terraform Diana-Montalion Hashicorp Learning-Systems-Thinking Ravi-Mishra Systems-Thinking Terraform-Certification Apache-Spark Brian-Paul Data-Engineering-With-Aws Etl Docker Blockchain-and-Banking-How-Technological-Innovations-Are-Shaping-the-Banking-Industry-Pierluigi-Martino Book-Review Boyan-Angelov Pierluigi Martino Python-and-R-for-the-Modern-Data-Scientist R-Programming Rick-Scavetta Apache-Nifi Kubernetes Rag Apache-Kafka Streaming Elasticsearch Production Banking Data-Lake Machine-Learning S3 Classification Data-Governance Data-Visualization Hadoop Aws-Glue Bigquery Clustering Computational-Thinking Context-Engineering Cpp-Programmers-Mindset Data-Quality Data-Warehouse K-Means Kafka Langgraph Nifi Openmp Pandas Postgresql Pyspark Real-Time Sql Apache-Flink Azure