Chapter 14: Data Processing with Apache Spark
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Streaming Data with Apache Kafka
Chapter 13 showed you how to move streaming data. Chapter 14 asks what you do when the volume is too big for a single Python script or a NiFi flow. Crickard’s answer is Apache Spark. Download it, stand up a small cluster on your laptop, and learn PySpark DataFrames.
What Spark is and why it exists
Spark is a distributed processing engine. It handles batch data, streaming data, and even graphs. The book focuses on the core: run computations across multiple machines so transformations that would choke one box finish in reasonable time.
In production you would run Spark on YARN, Kubernetes, Mesos, or EMR. For learning, Crickard uses standalone mode. Same concepts, less infrastructure pain.
He downloads Spark 3.0.0 (pre-built for Hadoop 2.7), extracts it to ~/spark3, copies the directory to ~/spark-node to simulate a second machine, and renames the start scripts from master/slave to head/node. Small detail, but he explains why. The old terminology is being phased out across the industry.
Start the head node, start the worker pointing at spark://hostname:7077, browse to localhost:8080, and you have a two-node cluster. The web UI shows running and completed applications. You will stare at this page a lot while debugging.
PySpark setup
PySpark ships with Spark. You set three environment variables:
SPARK_HOMEpointing at your install- Add
$SPARK_HOME/binto PATH PYSPARK_PYTHON=python3
Drop those in ~/.bashrc so they survive terminal restarts.
For Jupyter notebooks, two options: set PYSPARK_DRIVER_PYTHON to launch notebooks directly, or use findspark (pip3 install findspark) and call findspark.init() at the top of each notebook. The book uses findspark. Two lines and you are connected.
Pi estimation: the hello world
Before real data work, Crickard runs Spark’s Pi estimation example. Boilerplate every Spark app shares:
findspark.init()- Create a
SparkSessionwith.master('spark://hostname:7077')and anappName - Do the work
spark.stop()
The pi code parallelizes random points across the cluster, filters those inside a unit circle, counts them, and estimates pi. You can see the job running in the web UI under your app name. Not useful for data engineering, but it proves the cluster works.
Spark DataFrames vs pandas
The real content is DataFrame manipulation. If you know pandas, Spark feels familiar with different syntax.
Read CSV: spark.read.csv('data.csv') instead of pd.read_csv(). First attempt gives you _c0 column names and all strings because Spark did not infer headers or types. Fix it with header=True, inferSchema=True.
View data: .show() instead of printing the DataFrame. Check types: .printSchema() instead of .dtypes.
Select and filter work similarly but with method chaining:
df.select('name').show()
df.filter('age<40').select(['name','age','state']).show()
Collect rows with .collect() when you need to iterate in Python. Convert a row to a dict with .asDict() and access fields by key.
Prefer SQL? Create a temp view and query it:
df.createOrReplaceTempView('people')
spark.sql('select * from people where age > 40').show()
Same results, different syntax. Pick what your team knows.
Aggregations and functions
describe('age') gives count, mean, stddev, min, max. groupBy('state').count() tallies records per state. agg({'age':'mean'}) computes column-level aggregates.
The pyspark.sql.functions module (imported as f) adds more tools: collect_set, countDistinct, md5, reverse, soundex. Crickard demos a handful. The API docs have hundreds more.
When done, always spark.stop(). Orphaned sessions clutter the cluster UI and eat memory.
What I think
This chapter is setup-heavy and concept-light compared to the NiFi chapters. That is fair. Spark has a bigger install footprint and more moving parts. Once you are running, the DataFrame section moves fast.
The pandas comparison helps if you already know pandas. If you do not, you might feel rushed. Spark’s lazy evaluation and distributed nature are barely mentioned. A newcomer might wonder why .show() is needed everywhere or why collect() on a huge DataFrame is dangerous.
Spark 3.0.0 and Hadoop 2.7 bundles are old. Current Spark runs fine without Hadoop on Linux. The head/node rename is optional now since upstream scripts have been updated. But walking through cluster setup on one machine still teaches you what the web UI shows and how workers register.
What is missing: structured streaming (the natural follow-up to Chapter 13’s Kafka work), writing DataFrames back to Kafka or JDBC, and any discussion of partitioning strategy for performance. The chapter title promises data processing and delivers DataFrame basics. Good foundation, not the full story.
Key takeaway
Spark spreads work across nodes. PySpark gives you a pandas-like API on top. The pattern is always the same: init, create session, transform, stop. For data engineering at scale, this is the tool behind the NiFi and Kafka layers you already built.
Previous: Streaming Data with Apache Kafka | Next: Real-Time Edge Data with MiNiFi, Kafka, and Spark