Data Engineering with Python: A Hands-On Book Walkthrough
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I’m starting a walkthrough of Data Engineering with Python by Paul Crickard. This is not a copy-paste summary. It’s my retelling of each chapter with notes on what stuck, what felt dated, and what still holds up.
What this book is about
Crickard teaches data engineering through building. You install real tools on a Linux machine. You write Python. You drag processors around in Apache NiFi. You schedule jobs in Airflow. You load data into PostgreSQL and Elasticsearch. By the end you have batch pipelines, production patterns, and a streaming stack with Kafka and Spark.
The subtitle says it plainly: work with large datasets, design data models, and automate pipelines using Python.
Why it matters
Data science gets the spotlight. Data engineering keeps the lights on. Someone has to extract data from messy sources, clean it, move it to the right place, and make sure it runs again tomorrow without breaking.
This book fills a gap that still exists. Many tutorials stop at a Jupyter notebook. Crickard pushes you toward operational thinking: staging tables, idempotent runs, version control for pipelines, monitoring, and deployment.
How the book is structured
Three sections, fifteen chapters, one appendix:
- Building data pipelines (ETL): Chapters 1-6 cover fundamentals, infrastructure setup, files, databases, data cleaning, and a full 311 service-request pipeline project.
- Deploying in production: Chapters 7-11 add staging, atomicity, NiFi Registry, monitoring, deployment, and a production-grade rebuild of the 311 project.
- Beyond batch: Chapters 12-15 introduce Kafka, Spark, and MiNiFi for streaming and edge data collection.
The appendix covers NiFi clustering.
Who should read along
The book targets analysts, ETL developers, and people moving into data engineering. You need basic Python and enough Linux to run terminal commands. No prior data engineering experience required.
Fair warning: the stack is from 2020. NiFi 1.x, older Kafka and Spark versions, MiNiFi 0.5.0. The concepts transfer. Some install steps and version pins will need updating.
What I’ll do in this series
Each post retells one chapter. I’ll explain the main ideas in plain language, call out useful examples, and share honest reactions. Where the book shows code or config, I’ll describe what it’s doing without dumping entire blocks.
If you’re learning data engineering and want a guided tour before (or while) you read the book, this series is for you.