Chapter 11: Three AWS Data Engineering Case Studies
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Streaming
Chapter 15: Real-Time Edge Data With MiNiFi, Kafka, and Spark
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Data Processing with Apache Spark
The last chapter of the main book tackles a problem the earlier NiFi chapters do not: what happens when the data source is a Raspberry Pi, a factory sensor, or some other small device that cannot run full NiFi? Crickard’s answer is Apache MiNiFi. Lightweight agent on the edge, full NiFi back at headquarters.
Chapter 13: Streaming Data With Apache Kafka
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Building a Kafka Cluster
Chapter 12 got Kafka running on your machine. Chapter 13 is where it starts to click. Crickard explains what Kafka actually is under the hood, wires it into the NiFi pipelines you built earlier, and shows you how to produce and consume messages with Python.
Building a Three-Node Apache Kafka Cluster
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Section 3 of the book shifts from batch to streaming. Chapter 12 is infrastructure first: build a real Kafka cluster, not a single-broker tutorial that leaves you guessing how production works.
Chapter 5: Real-Time Data With Amazon Kinesis
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Batch vs Streaming Data Pipelines and Windowing
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 7 is where the book stops talking about storage and starts talking about movement. Data pipelines are the plumbing. They collect, ingest, process, store, and serve data. Sometimes on a schedule, sometimes the second an event happens.
Streaming Data With Apache Kafka - Study Notes From Data Engineering With Python Ch 13
Up to this point in the book, data pipelines have been about moving data that already exists. Query a database, read a file, process it, store it. The data sits still and you go get it.
Building a Kafka Cluster - Study Notes From Data Engineering With Python Ch 12
Up to this point in the book, everything has been batch processing. You query a database, get a full dataset, transform it, load it somewhere. The data sits still while you work on it.
Stream Processing With Apache Flink: True Real-Time Analytics
Previous: Flink DataSet API: Transformations, Joins, and Aggregations
We’ve talked about how Spark handles streaming using micro-batches. It’s a great approach, but some people argue it’s not “true” streaming. If you need the absolute lowest latency possible, you want Apache Flink.