Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 7 is where the book stops talking about storage and starts talking about movement. Data pipelines are the plumbing. They collect, ingest, process, store, and serve data. Sometimes on a schedule, sometimes the second an event happens.
This is Part 1 of Chapter 7. Part 2 covers orchestration and transformations.
Chapter 7 of Data Engineering for Beginners is probably where things start feeling real. You stop talking about storage and tables and start talking about how data actually moves. And the answer is: through pipelines.
This is post 9 in my Data Engineering with AWS retelling series.
You have your whiteboard architecture from Chapter 5. You know who your data consumers are and what they need. Now it is time to actually move data. Chapter 6 covers data ingestion – getting data from wherever it lives into your AWS data lake. This first part focuses on batch ingestion from databases and files. Part 2 covers streaming.
Previous: Structured Streaming: The Modern Way to Handle Data Streams
We’ve spent a lot of time on Spark, and for good reason - it’s amazing. But if you’re serious about big data, you need to know about Apache Flink. In Chapter 8, Sridhar Alla introduces us to the technology that many experts consider the “true” successor to MapReduce for real-time processing.