Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Streaming Data with Apache Kafka
Chapter 13 showed you how to move streaming data. Chapter 14 asks what you do when the volume is too big for a single Python script or a NiFi flow. Crickard’s answer is Apache Spark. Download it, stand up a small cluster on your laptop, and learn PySpark DataFrames.
In the last post, we looked at Spark’s architecture. Now, let’s talk about how you actually write code for it. Neylson Crepalde highlights two main ways to interact with Spark: the DataFrame API and Spark SQL.
If there is one tool that defined the “Big Data” era, it’s Apache Spark. It’s the engine that handles everything from terabyte-scale ETL to complex machine learning. In Chapter 5, Neylson Crepalde breaks down exactly how Spark works and why it’s so powerful on Kubernetes.
You have streaming data. You have batch data. You have a lot of it. Now you need to actually process it. Fast. On more than one machine.
In Part 1 we set up a Dataproc cluster, got familiar with HDFS, and touched on what a data lake actually is. Now it is time to get into the real work: writing PySpark code, understanding RDDs, moving data between HDFS, GCS, and BigQuery, and learning how to actually submit Spark jobs to Dataproc.