Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Streaming Data with Apache Kafka
Chapter 13 showed you how to move streaming data. Chapter 14 asks what you do when the volume is too big for a single Python script or a NiFi flow. Crickard’s answer is Apache Spark. Download it, stand up a small cluster on your laptop, and learn PySpark DataFrames.
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1)
At some point, one CPU with a handful of threads stops being enough. Chapter 13 is Morley’s tour of what changes when you scale up: more cores, more processes, more machines. The rubber duckies app fit on a laptop. Real workloads often do not.
If there is one tool that defined the “Big Data” era, it’s Apache Spark. It’s the engine that handles everything from terabyte-scale ETL to complex machine learning. In Chapter 5, Neylson Crepalde breaks down exactly how Spark works and why it’s so powerful on Kubernetes.