Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I finished the full walkthrough. Fifteen chapters, one appendix, two big projects, and a lot of terminal time. Here’s where I landed.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Real-Time Edge Data with MiNiFi, Kafka, and Spark
The appendix is a bonus chapter on NiFi clustering. By this point you have Kafka, ZooKeeper, and Spark running across pseudo-nodes on one machine. Crickard extends the same idea to NiFi: spread pipeline work across multiple servers instead of buying a bigger box.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Data Processing with Apache Spark
The last chapter of the main book tackles a problem the earlier NiFi chapters do not: what happens when the data source is a Raspberry Pi, a factory sensor, or some other small device that cannot run full NiFi? Crickard’s answer is Apache MiNiFi. Lightweight agent on the edge, full NiFi back at headquarters.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Streaming Data with Apache Kafka
Chapter 13 showed you how to move streaming data. Chapter 14 asks what you do when the volume is too big for a single Python script or a NiFi flow. Crickard’s answer is Apache Spark. Download it, stand up a small cluster on your laptop, and learn PySpark DataFrames.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Building a Kafka Cluster
Chapter 12 got Kafka running on your machine. Chapter 13 is where it starts to click. Crickard explains what Kafka actually is under the hood, wires it into the NiFi pipelines you built earlier, and shows you how to produce and consume messages with Python.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Section 3 of the book shifts from batch to streaming. Chapter 12 is infrastructure first: build a real Kafka cluster, not a single-broker tutorial that leaves you guessing how production works.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Chapter 11 is the capstone for the production section. Crickard takes everything from Chapters 7 through 10 and builds a real pipeline: read JSON files from a data lake, load staging, validate, load warehouse, version every piece, deploy to production with variables.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Software teams have dev, test, and production environments. Data engineering should work the same way. You build on one machine, prove it works somewhere that looks like prod, then promote it. Chapter 10 is about making that promotion less painful in NiFi.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
You built a pipeline that is idempotent, version controlled, and ready for production. Great. But production means things break when you are not looking. Elasticsearch goes down. A network hiccup kills a connection. Bad data slips through. Chapter 9 is about knowing when that happens without staring at a screen all day.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
You can build a beautiful NiFi pipeline and still be one bad drag-and-drop away from losing hours of work. Chapter 8 fixes that gap. Version control for data pipelines is not optional in production. Crickard treats data engineers like software engineers here, which is the right framing. You would not write application code without Git. You should not run production flows without versioning either.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Section 2 of the book starts here. The vibe shifts from “learn the tools” to “do not break production.” Chapter 7 covers three ideas that separate hobby pipelines from ones you can run at 3 a.m. without panic: staging, validation, idempotency, and atomicity. Crickard uses a fake “Widget Co” sales example and a people-data pipeline he actually runs, though he admits that one is low-stakes and missing pieces a critical system would need.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 6 is where the book stops feeling like a tutorial and starts feeling like a real project. Crickard builds a pipeline that pulls citizen service requests from SeeClickFix (think 311-style complaints: graffiti, potholes, abandoned cars, needles, people ignoring social distancing during COVID). The data lands in Elasticsearch. Then you build a Kibana dashboard on top. He says he still runs this pipeline every 8 hours. That detail matters. This is not a throwaway demo.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapters 3 and 4 taught you how to move data in and out of files and databases. Chapter 5 asks the next obvious question: what do you do with the data in the middle? Often the answer is clean it, reshape it, and sometimes bolt on extra context from another source. Crickard uses real Albuquerque e-scooter trip data for all of this, which makes the examples feel grounded instead of toy-like.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 4 is where the book stops playing with files on disk and starts working with databases. For most data engineers, this is the daily bread.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 3 is where the book starts feeling like data engineering instead of systems administration.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 told you what data engineering is. Chapter 2 asks you to build the whole workshop on one Linux machine.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 of Paul Crickard’s book is the orientation lecture. No installs yet. No code. Just a clear picture of what data engineering is, who does it, and what tools show up again and again.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I’m starting a walkthrough of Data Engineering with Python by Paul Crickard. This is not a copy-paste summary. It’s my retelling of each chapter with notes on what stuck, what felt dated, and what still holds up.
That’s it. Fifteen chapters, seventeen posts, and one complete walkthrough of Paul Crickard’s Data Engineering with Python (Packt, 2020, ISBN: 978-1-83921-418-9).
You have NiFi running. Kafka is streaming. Spark is processing. But what about the data source? What happens when your data comes from a tiny sensor or a Raspberry Pi that can barely run a web browser?
You have streaming data. You have batch data. You have a lot of it. Now you need to actually process it. Fast. On more than one machine.
Up to this point in the book, data pipelines have been about moving data that already exists. Query a database, read a file, process it, store it. The data sits still and you go get it.
Up to this point in the book, everything has been batch processing. You query a database, get a full dataset, transform it, load it somewhere. The data sits still while you work on it.
You learned the individual tools. You learned the deployment strategies. Now Chapter 11 of Data Engineering with Python by Paul Crickard puts it all together. This is the chapter where you build a complete, production-grade data pipeline from start to finish.
You built your data pipelines. They work on your laptop. Now what? Chapter 10 of Data Engineering with Python by Paul Crickard covers the part everyone eventually has to face: getting your pipelines out of development and into production.
You built a data pipeline. It is idempotent, uses atomic transactions, and has version control. It is production ready. But can you tell when it breaks?
You’ve been building data pipelines for several chapters now. They work. They move data. But here’s the problem: none of them have version control. If you break something, there’s no going back. Chapter 8 of Data Engineering with Python by Paul Crickard fixes that. It introduces the NiFi Registry, a sub-project of Apache NiFi that handles version control for your data pipelines.
You built a pipeline. It works on your machine. It runs on a schedule. Data goes in, data comes out. Ship it, right?
The previous chapters taught you the individual tools. Python, NiFi, Airflow, databases, data cleaning. Chapter 6 of Data Engineering with Python by Paul Crickard puts them all together into one real project.
You can build the best pipeline in the world. You can read files, write to databases, schedule everything with Airflow. But if the data going through that pipeline is messy, none of it matters.
Most data pipelines start with a database. Most of them end with one too. Chapter 4 of Paul Crickard’s book is about connecting Python to databases and moving data between them. If the previous chapter was about flat files, this one is where things get real.
Chapter 3 is where Crickard moves from setup to actual work. You installed all those tools in Chapter 2. Now you use them. The chapter covers one of the most fundamental tasks in data engineering: getting data out of text files and into something useful.
Chapter 1 was all theory. Now it’s time to actually install stuff. Chapter 2 of Data Engineering with Python by Paul Crickard is a setup chapter. You install the tools, configure them, and make sure everything talks to each other.
Chapter 1 of Data Engineering with Python by Paul Crickard starts with the basics. What is data engineering? What do data engineers actually do? And how is it different from data science?
So I picked up Data Engineering with Python by Paul Crickard (Packt, 2020, ISBN: 978-1-83921-418-9) and decided to write up my study notes as I go through it. I’ve been working in IT for over 20 years, and data engineering keeps coming up everywhere. This book seemed like a good one to work through and share what I learn.