Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
The main chapters end at GCP. Three appendices handle the practical stuff: how to run the code, how imports changed in Airflow 2, and how to map Airflow metrics for Prometheus.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Operating Airflow in Production (Part 1) | Next: Securing Apache Airflow
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
You built a pipeline that is idempotent, version controlled, and ready for production. Great. But production means things break when you are not looking. Elasticsearch goes down. A network hiccup kills a connection. Bad data slips through. Chapter 9 is about knowing when that happens without staring at a screen all day.
You built a data pipeline. It is idempotent, uses atomic transactions, and has version control. It is production ready. But can you tell when it breaks?