Adi Wijaya's practical guide to building scalable data pipelines and platforms using Google Cloud Platform services like BigQuery, Dataproc, Dataflow, and Cloud Composer.
Data Engineering with Google Cloud Platform (2nd edition, 2024) walks you through the full stack of data engineering on GCP. It starts with fundamentals like ETL, data warehouses, and data lakes, then moves to hands-on building with BigQuery, Cloud Composer (Airflow), Dataproc (Spark), Pub/Sub, Dataflow (Beam), Looker Studio, and Vertex AI.
The book is split into three parts. Part one covers data engineering basics and the GCP ecosystem. Part two is the bulk of the book, where you build actual pipelines, data warehouses, data lakes, streaming systems, visualizations, and ML workflows. Part three tackles the strategic side: project management, data governance, cost control, CI/CD, and career growth including GCP certification prep.
Written by a cloud data engineer at Google with over a decade of experience, the book targets aspiring data engineers, people preparing for the GCP Professional Data Engineer certification, and teams migrating data workloads to Google Cloud. The second edition updates coverage to include Dataform, Dataproc Serverless, BigQuery editions pricing, and Vertex AI pipelines.
I just finished reading “Data Engineering with Google Cloud Platform” by Adi Wijaya (2nd edition, Packt Publishing, 2024) and I want to share what I learned. Not as a dry summary, but more like telling a friend what the book is about over coffee.
Chapter 1 starts with a confession most of us in the data world can relate to. Adi Wijaya says he used to think data was clean. Neatly organized, ready to go. Then he actually worked with data in real organizations and realized most of the effort goes into collecting, cleaning, and transforming it. Not the fun machine learning part. The plumbing part.
Chapter 2 is where Adi Wijaya starts showing what Google Cloud Platform actually has for data engineers. After the theory in Chapter 1, this one is about opening GCP for the first time and figuring out which services matter and which ones you can safely ignore for now.
Chapter 3 is where things get real. Up to now the book was setting the stage, explaining what data engineering is, showing you around GCP. Now Adi Wijaya says: okay, let’s actually build something. And the something is a data warehouse in BigQuery.
In Part 1 we loaded CSV files into BigQuery and built a simple warehouse from a MySQL export. Now the book throws a second scenario at us: bike-sharing data. More tables, daily batch loading, and a real question that every data engineer has to face sooner or later. How do you actually model your data so that business people can use it without calling you every five minutes?
Up until now in the book, we built BigQuery tables by hand, wrote queries in the console, and loaded data manually. That works for learning, but nobody does that in production. In production, you need things to run on their own, on schedule, without you babysitting them at 5 AM.
In the first part we got Cloud Composer running, wrote our first DAGs, and learned operators. This second part covers the stuff that separates beginner Airflow code from production-ready pipelines: variables, idempotent tasks, backfilling, sensors, and dataset-driven scheduling.
Chapter 5 is where things get interesting if you come from a traditional database background. We are leaving the nice structured world of BigQuery and entering the territory of raw files, distributed storage, and the Hadoop ecosystem. Welcome to the data lake.
In Part 1 we set up a Dataproc cluster, got familiar with HDFS, and touched on what a data lake actually is. Now it is time to get into the real work: writing PySpark code, understanding RDDs, moving data between HDFS, GCS, and BigQuery, and learning how to actually submit Spark jobs to Dataproc.
Chapter 6 is where Adi Wijaya switches gears from batch to real-time. After spending Chapters 3 through 5 on batch pipelines with BigQuery, Cloud Composer, and Dataproc, now it is time to talk about streaming data. Two GCP services carry this chapter: Pub/Sub and Dataflow. This post covers the streaming concepts and Pub/Sub. Dataflow gets its own post in Part 2.
In Part 1 we covered Pub/Sub and how messages flow between publishers and subscribers. Now comes the fun part: what do you actually do with those messages once you have them? That’s where Apache Beam and Dataflow come in.
You spend weeks building pipelines, modeling data, setting up orchestration. Everything works. Data lands in BigQuery clean and on time. And then someone from the business side asks: “So… where do I see the numbers?” That is exactly where Chapter 7 picks up. All that upstream work has to end somewhere useful, and for most organizations that somewhere is a dashboard.
Chapter 8 is the one where Adi Wijaya finally brings up the topic every data engineer either loves or dreads: machine learning. And honestly, he does a good job of calming down both camps. If you are excited about ML, great. If you think it has nothing to do with your job, think again. This chapter shows why ML and data engineering are way closer than most people realize.
Part 1 covered the ML basics: what supervised and unsupervised learning are, how a simple model gets trained, and why data engineers should care about ML at all. Now in Part 2, Adi Wijaya moves into the GCP tools that make ML actually work in production. This is where theory meets infrastructure.
Chapter 9 is the one where Adi Wijaya zooms out from data pipelines and asks: okay, but who can access what, and how do we keep this whole thing organized? If the previous chapters taught you how to build things in GCP, this one teaches you how to not let those things turn into a security and management mess.
Data governance is one of those topics that sounds boring until you realize nobody can find anything in your data platform. Then it becomes very interesting very fast.
In Part 1 we covered how data governance breaks into three pillars (usability, security, accountability) and went through metadata, Dataplex search, access control in BigQuery, and the Sensitive Data Protection service for finding PII. Now let’s pick up where we left off: understanding what SDP actually finds, and then moving into the accountability pillar.
Nobody ever got promoted for building the cheapest data pipeline. But plenty of people have gotten uncomfortable phone calls from their CFO after a runaway BigQuery bill. Chapter 11 is about the money side of GCP, and I think this is one of the most practical chapters in the book.
Chapter 12 is a shift from everything we have done so far. Until now, we were learning how to build things: pipelines, data lakes, warehouses, streaming systems. Now the question is: how do you ship all that stuff to production without breaking things? The answer is CI/CD.
In Part 1 we covered the theory behind CI/CD and ran through a basic Cloud Build exercise with a Python project. Unit tests ran automatically on every push. Broken code got caught before it reached production. Good stuff, but that was a simple calculator script. Now we need to connect this to real data engineering work.
Chapter 13 is the last chapter in the book, and it’s different from everything that came before. No new GCP services, no hands-on exercises, no Terraform scripts. Instead, Adi Wijaya steps back and talks about the bigger picture: certifications, where data engineering is heading, and how to actually feel confident in this role.
In Part 1 we went through the quiz questions, extra GCP services, and how the book ties everything together. Now let’s talk about the stuff that matters after you close the book: getting certified, planning your career, and figuring out what comes next.
Twenty-two posts later, we are done. This was my retelling of “Data Engineering with Google Cloud Platform” by Adi Wijaya (2nd edition, Packt Publishing, 2024, ISBN 978-1-83508-011-5). Time to look back, share what stuck with me, and give an honest assessment.