Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
The main chapters end at GCP. Three appendices handle the practical stuff: how to run the code, how imports changed in Airflow 2, and how to map Airflow metrics for Prometheus.
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Testing Airflow DAGs (Part 2) | Next: Running Airflow Tasks in Containers (Part 2)
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Testing Airflow DAGs (Part 1) | Next: Running Airflow Tasks in Containers (Part 1)
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Triggering Airflow Workflows | Next: Building Custom Airflow Components (Part 1)
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 1 was theory. Chapter 2 is hands-on. The book follows John, a rocket enthusiast who wants upcoming launch images on his machine. The Launch Library API returns JSON with image URLs. John’s plan: download metadata, fetch pictures, notify when done. That becomes your first real Airflow DAG.
In my last post, we talked about why containers are the bedrock of modern data engineering. But honestly, just running other people’s images only gets you so far. The real magic happens when you start packaging your own custom code.
If you are working with data today, you can’t really ignore containers. They have become the standardized unit for how we develop, ship, and deploy software. But why do we care so much about them in the big data world?
You built your data pipelines. They work on your laptop. Now what? Chapter 10 of Data Engineering with Python by Paul Crickard covers the part everyone eventually has to face: getting your pipelines out of development and into production.
In Part 1 we built a FastAPI app, Dockerized it, and locked it down with API keys. That is the “maximum control” path. It works great, but it requires a lot of boilerplate. Part 2 covers two things: automating the whole thing with CI/CD, and a completely different approach that makes most of that boilerplate disappear.
Chapter 7 of Laura Funderburk’s book is where the rubber meets the road. You built a RAG pipeline in Chapter 6. Now you need to ship it. Get it out of a notebook and into something that real users can hit with HTTP requests.
In Part 1, we covered how Funderburk moves from Jupyter notebooks to a production-ready project structure. Docker, uv, SuperComponents, dual Elasticsearch. Now comes the part that actually tells you if your RAG pipeline is any good: systematic evaluation with RAGAS and continuous monitoring with Weights and Biases.
Chapter 6 is where the book shifts gears. Hard. Funderburk basically says: “Cool, you built a RAG pipeline. It works on your laptop. Now what?”
Chapter 1 was all theory. Now it’s time to actually install stuff. Chapter 2 of Data Engineering with Python by Paul Crickard is a setup chapter. You install the tools, configure them, and make sure everything talks to each other.