Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
GCP is the last cloud chapter and the most complete. Google has the largest Airflow provider package. The book walks through a full GKE deployment with Helm, not just diagrams. This is DIY Airflow, not Cloud Composer (covered in chapter 15).
You spend weeks building pipelines, modeling data, setting up orchestration. Everything works. Data lands in BigQuery clean and on time. And then someone from the business side asks: “So… where do I see the numbers?” That is exactly where Chapter 7 picks up. All that upstream work has to end somewhere useful, and for most organizations that somewhere is a dashboard.
In Part 1 we loaded CSV files into BigQuery and built a simple warehouse from a MySQL export. Now the book throws a second scenario at us: bike-sharing data. More tables, daily batch loading, and a real question that every data engineer has to face sooner or later. How do you actually model your data so that business people can use it without calling you every five minutes?
Chapter 3 is where things get real. Up to now the book was setting the stage, explaining what data engineering is, showing you around GCP. Now Adi Wijaya says: okay, let’s actually build something. And the something is a data warehouse in BigQuery.
I just finished reading “Data Engineering with Google Cloud Platform” by Adi Wijaya (2nd edition, Packt Publishing, 2024) and I want to share what I learned. Not as a dry summary, but more like telling a friend what the book is about over coffee.