Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Real-Time Edge Data with MiNiFi, Kafka, and Spark
The appendix is a bonus chapter on NiFi clustering. By this point you have Kafka, ZooKeeper, and Spark running across pseudo-nodes on one machine. Crickard extends the same idea to NiFi: spread pipeline work across multiple servers instead of buying a bigger box.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Data Processing with Apache Spark
The last chapter of the main book tackles a problem the earlier NiFi chapters do not: what happens when the data source is a Raspberry Pi, a factory sensor, or some other small device that cannot run full NiFi? Crickard’s answer is Apache MiNiFi. Lightweight agent on the edge, full NiFi back at headquarters.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Building a Kafka Cluster
Chapter 12 got Kafka running on your machine. Chapter 13 is where it starts to click. Crickard explains what Kafka actually is under the hood, wires it into the NiFi pipelines you built earlier, and shows you how to produce and consume messages with Python.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
You can build a beautiful NiFi pipeline and still be one bad drag-and-drop away from losing hours of work. Chapter 8 fixes that gap. Version control for data pipelines is not optional in production. Crickard treats data engineers like software engineers here, which is the right framing. You would not write application code without Git. You should not run production flows without versioning either.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 6 is where the book stops feeling like a tutorial and starts feeling like a real project. Crickard builds a pipeline that pulls citizen service requests from SeeClickFix (think 311-style complaints: graffiti, potholes, abandoned cars, needles, people ignoring social distancing during COVID). The data lands in Elasticsearch. Then you build a Kibana dashboard on top. He says he still runs this pipeline every 8 hours. That detail matters. This is not a throwaway demo.