A hands-on guide to building batch and streaming data pipelines with Python, Apache NiFi, Airflow, Kafka, and Spark.
Data Engineering with Python by Paul Crickard (Packt, 2020) teaches data engineering by doing. Instead of staying in theory, you install a full stack on Linux: Apache NiFi, Airflow, PostgreSQL, Elasticsearch, Kibana, Kafka, Spark, and MiNiFi. You write Python for extraction, transformation, validation, and monitoring. You build two end-to-end projects: a 311 service-request pipeline with a Kibana dashboard, then a production-hardened version with staging, data quality checks, versioning, and deployment workflows.
The book opens with fundamentals. What do data engineers actually do? How is the role different from data science? What does ETL look like in a growing business? From there it moves into infrastructure setup and core skills: reading CSV and JSON files, loading relational and NoSQL databases, cleaning messy data with pandas, and orchestrating workflows in Airflow and NiFi.
Section two shifts to production concerns. You learn staging patterns, idempotent and atomic pipeline design, version control with the NiFi Registry, monitoring through the NiFi GUI and REST API, and strategies for promoting pipelines from test to production. Section three goes beyond batch processing with a Kafka cluster, Python producers and consumers, PySpark data processing, and MiNiFi for edge and IoT data collection.
The writing is practical and example-driven. Crickard comes from a public-sector data background, and the 311 project grounds the material in real civic data. Some tool versions are dated since the 2020 release, but the operational patterns (validate before load, version your pipelines, monitor failures, separate environments) remain directly applicable. Best for analysts, ETL developers, or backend engineers moving into data engineering who want a broad, build-it-yourself introduction.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I’m starting a walkthrough of Data Engineering with Python by Paul Crickard. This is not a copy-paste summary. It’s my retelling of each chapter with notes on what stuck, what felt dated, and what still holds up.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 of Paul Crickard’s book is the orientation lecture. No installs yet. No code. Just a clear picture of what data engineering is, who does it, and what tools show up again and again.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 told you what data engineering is. Chapter 2 asks you to build the whole workshop on one Linux machine.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 3 is where the book starts feeling like data engineering instead of systems administration.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 4 is where the book stops playing with files on disk and starts working with databases. For most data engineers, this is the daily bread.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapters 3 and 4 taught you how to move data in and out of files and databases. Chapter 5 asks the next obvious question: what do you do with the data in the middle? Often the answer is clean it, reshape it, and sometimes bolt on extra context from another source. Crickard uses real Albuquerque e-scooter trip data for all of this, which makes the examples feel grounded instead of toy-like.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 6 is where the book stops feeling like a tutorial and starts feeling like a real project. Crickard builds a pipeline that pulls citizen service requests from SeeClickFix (think 311-style complaints: graffiti, potholes, abandoned cars, needles, people ignoring social distancing during COVID). The data lands in Elasticsearch. Then you build a Kibana dashboard on top. He says he still runs this pipeline every 8 hours. That detail matters. This is not a throwaway demo.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Section 2 of the book starts here. The vibe shifts from “learn the tools” to “do not break production.” Chapter 7 covers three ideas that separate hobby pipelines from ones you can run at 3 a.m. without panic: staging, validation, idempotency, and atomicity. Crickard uses a fake “Widget Co” sales example and a people-data pipeline he actually runs, though he admits that one is low-stakes and missing pieces a critical system would need.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
You can build a beautiful NiFi pipeline and still be one bad drag-and-drop away from losing hours of work. Chapter 8 fixes that gap. Version control for data pipelines is not optional in production. Crickard treats data engineers like software engineers here, which is the right framing. You would not write application code without Git. You should not run production flows without versioning either.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
You built a pipeline that is idempotent, version controlled, and ready for production. Great. But production means things break when you are not looking. Elasticsearch goes down. A network hiccup kills a connection. Bad data slips through. Chapter 9 is about knowing when that happens without staring at a screen all day.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Software teams have dev, test, and production environments. Data engineering should work the same way. You build on one machine, prove it works somewhere that looks like prod, then promote it. Chapter 10 is about making that promotion less painful in NiFi.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Chapter 11 is the capstone for the production section. Crickard takes everything from Chapters 7 through 10 and builds a real pipeline: read JSON files from a data lake, load staging, validate, load warehouse, version every piece, deploy to production with variables.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Section 3 of the book shifts from batch to streaming. Chapter 12 is infrastructure first: build a real Kafka cluster, not a single-broker tutorial that leaves you guessing how production works.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Building a Kafka Cluster
Chapter 12 got Kafka running on your machine. Chapter 13 is where it starts to click. Crickard explains what Kafka actually is under the hood, wires it into the NiFi pipelines you built earlier, and shows you how to produce and consume messages with Python.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Streaming Data with Apache Kafka
Chapter 13 showed you how to move streaming data. Chapter 14 asks what you do when the volume is too big for a single Python script or a NiFi flow. Crickard’s answer is Apache Spark. Download it, stand up a small cluster on your laptop, and learn PySpark DataFrames.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Data Processing with Apache Spark
The last chapter of the main book tackles a problem the earlier NiFi chapters do not: what happens when the data source is a Raspberry Pi, a factory sensor, or some other small device that cannot run full NiFi? Crickard’s answer is Apache MiNiFi. Lightweight agent on the edge, full NiFi back at headquarters.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Real-Time Edge Data with MiNiFi, Kafka, and Spark
The appendix is a bonus chapter on NiFi clustering. By this point you have Kafka, ZooKeeper, and Spark running across pseudo-nodes on one machine. Crickard extends the same idea to NiFi: spread pipeline work across multiple servers instead of buying a bigger box.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I finished the full walkthrough. Fifteen chapters, one appendix, two big projects, and a lot of terminal time. Here’s where I landed.