The second half of Chapter 4 in Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412) is where SQL goes from “filter a table” to “actually analyze data across tables.” This is the part that makes SQL feel like a real data engineering skill.
This is Part 2 of Chapter 4, continuing from the SQL basics.
In Part 1 we covered how to pull data from one table. Filter it, sort it, count it. But real databases have many tables. Customers in one, orders in another, products in a third. The interesting stuff happens when you combine them.
Previous: Batch Analytics with Apache Spark: Faster Than MapReduce
In the last post, we looked at why Spark is so fast. Today, we’re getting into the nitty-gritty of how to actually use it. If you’re a SQL fan, you’re going to love this. Chapter 6 of Sridhar Alla’s book spends a lot of time on Spark SQL, and for good reason - it’s where most of the work happens.