Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Software teams have dev, test, and production environments. Data engineering should work the same way. You build on one machine, prove it works somewhere that looks like prod, then promote it. Chapter 10 is about making that promotion less painful in NiFi.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Operating Airflow in Production (Part 1) | Next: Securing Apache Airflow
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Airflow Best Practices (Part 2) | Next: Operating Airflow in Production (Part 2)
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Section 2 of the book starts here. The vibe shifts from “learn the tools” to “do not break production.” Chapter 7 covers three ideas that separate hobby pipelines from ones you can run at 3 a.m. without panic: staging, validation, idempotency, and atomicity. Crickard uses a fake “Widget Co” sales example and a people-data pipeline he actually runs, though he admits that one is low-stakes and missing pieces a critical system would need.
You learned the individual tools. You learned the deployment strategies. Now Chapter 11 of Data Engineering with Python by Paul Crickard puts it all together. This is the chapter where you build a complete, production-grade data pipeline from start to finish.
You built a pipeline. It works on your machine. It runs on a schedule. Data goes in, data comes out. Ship it, right?
In Part 1, we covered how Funderburk moves from Jupyter notebooks to a production-ready project structure. Docker, uv, SuperComponents, dual Elasticsearch. Now comes the part that actually tells you if your RAG pipeline is any good: systematic evaluation with RAGAS and continuous monitoring with Weights and Biases.
Chapter 6 is where the book shifts gears. Hard. Funderburk basically says: “Cool, you built a RAG pipeline. It works on your laptop. Now what?”