Final Thoughts on Data Engineering with Python

Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)

I finished the full walkthrough. Fifteen chapters, one appendix, two big projects, and a lot of terminal time. Here’s where I landed.

What this book does well

It makes data engineering tangible. You don’t just read about ETL. You install NiFi, Airflow, PostgreSQL, Elasticsearch, Kafka, and Spark on a Linux box. You build pipelines that move real data. That hands-on loop is still the best way to learn this stuff.

It teaches operational thinking early. Chapter 7 onward pushes past “make it work once” into staging, validation, idempotency, atomicity, versioning, monitoring, and deployment. That’s the gap most tutorials skip. Crickard doesn’t.

The 311 project ties everything together. Chapter 6 builds a full pipeline from an API to a Kibana dashboard. Chapter 11 rebuilds it for production with a data lake, staging tables, Great Expectations checks, and proper deployment patterns. Seeing the same project evolve is worth the read.

Python stays in the picture. Even with heavy NiFi usage, you write Python for API calls, cleaning, database work, monitoring scripts, and Kafka producers/consumers. The book title is honest about that mix.

What aged (and how to handle it)

Published in 2020, some specifics are dated:

• Tool versions: NiFi 1.x, MiNiFi 0.5.0, older Kafka/Spark builds. Install commands and compatibility notes (like needing NiFi 1.9.0 for MiNiFi) need updating.
• NiFi-heavy workflow: Many teams today lean more on Airflow, dbt, or cloud-native services. The NiFi depth is still valuable for understanding flow-based ETL, but it’s not the default stack everywhere.
• Great Expectations: The validation patterns are right. The API may have shifted since 2020. Check current docs when you implement.
• Jython in NiFi: Still works, but feels clunky next to modern ExecuteScript patterns or external Python services.

None of this kills the book. The patterns (extract, stage, validate, load, monitor, version, deploy) are what you carry forward.

Key takeaways

1. Data engineering is plumbing plus judgment. Moving data is the floor. Knowing when to stage, how to make runs repeatable, and what “production ready” means is the ceiling.

2. ETL is not one tool. Crickard uses Python, NiFi, Airflow, SQL databases, Elasticsearch, Kafka, and Spark in the same workflow. Real pipelines look like that.

3. Batch and streaming share a foundation. The book starts with batch file and database work, then extends into Kafka and Spark. The mental model transfers: sources, transforms, sinks, monitoring.

4. Production is a feature set, not a deployment step. Version control, environment separation, backpressure, alerting, and rollback plans are part of the pipeline design. Chapter 7-11 hammer this home.

5. Edge data is a real use case. MiNiFi for IoT and low-resource devices is a niche topic most books ignore. Short chapter, practical angle.

Who should read it

Good fit:

• Analysts or backend devs moving into data engineering
• People who learn by building on a VM
• Teams evaluating NiFi or wanting a broad stack tour
• Anyone who needs the “what happens after the notebook” picture

Maybe skip or skim:

• You already run mature pipelines on AWS/GCP with managed services only
• You want a deep single-tool focus (dbt-only, Spark-only, etc.)
• You cannot spare time for local cluster setup

My overall rating

I’d give it a solid 7.5/10 in 2026. The concepts score higher than the version pins. As a first data engineering book with Python in the title, it delivers more practical ops content than most competitors from the same era.

If you read along with this series, you have the map. If you work through the exercises yourself, you’ll have the muscle memory too.

The full series

1. Series Intro
2. What Is Data Engineering?
3. Building Data Engineering Infrastructure
4. Reading and Writing Files
5. Working with Databases
6. Cleaning, Transforming, and Enriching Data
7. Building a 311 Data Pipeline
8. Features of a Production Pipeline
9. NiFi Registry Version Control
10. Monitoring Data Pipelines
11. Deploying Data Pipelines
12. Building a Production Data Pipeline
13. Building a Kafka Cluster
14. Streaming Data with Apache Kafka
15. Data Processing with Apache Spark
16. Real-Time Edge Data with MiNiFi
17. NiFi Clustering Appendix

Previous: NiFi Clustering Appendix