Data Science: The Hard Parts by Daniel Vaughan (ISBN 978-1-098-14647-4)
← Previous: LLMs and the Future of Data Science
Seventeen chapters. Two parts. One preface that sets honest expectations. This series retold Daniel Vaughan’s Data Science: The Hard Parts chapter by chapter. Time to step back and say what stuck.
Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412, Wiley 2025)
Previous: Interview Questions Appendix
Thirteen chapters, an interview appendix, and a glossary later, here is where Data Engineering for Beginners lands.
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1)
Fifteen chapters, one rubber duckies mystery, and a lot of code later, here is where The C++ Programmer’s Mindset lands.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Twenty-five posts later, I have read the whole book. Here is where I landed.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I finished the full walkthrough. Fifteen chapters, one appendix, two big projects, and a lot of terminal time. Here’s where I landed.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412, Wiley 2025)
Previous: Building a Data Engineering Career
The appendix is a flashcard deck for interview prep. Nwokwu organized sample questions across five areas that match the interview stages from Chapter 13. I will retell the highlights here. Use this as a self-test before your next round.
Data Science: The Hard Parts by Daniel Vaughan (ISBN 978-1-098-14647-4)
← Previous: A/B Tests and Experimentation | Next: Series Closing
Chapter 17 is different. No techniques. No code snippets. Just speculation about what LLMs mean for data science.
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1)
You picked C++ partly for speed. Chapter 15 is Morley’s case for measuring before you touch anything. Premature optimization wastes time and adds complexity. Profiling tells you where speed actually matters and what is holding you back.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
The main chapters end at GCP. Three appendices handle the practical stuff: how to run the code, how imports changed in Airflow 2, and how to map Airflow metrics for Prometheus.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Real-Time Edge Data with MiNiFi, Kafka, and Spark
The appendix is a bonus chapter on NiFi clustering. By this point you have Kafka, ZooKeeper, and Spark running across pseudo-nodes on one machine. Crickard extends the same idea to NiFi: spread pipeline work across multiple servers instead of buying a bigger box.
Data Science: The Hard Parts by Daniel Vaughan (ISBN 978-1-098-14647-4)
← Previous: Incrementality and Causal Inference | Next: LLMs and the Future of Data Science
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
GCP is the last cloud chapter and the most complete. Google has the largest Airflow provider package. The book walks through a full GKE deployment with Helm, not just diagrams. This is DIY Airflow, not Cloud Composer (covered in chapter 15).
Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412, Wiley 2025)
Previous: Cloud Services and Cost
Chapter 13 is the career chapter. Nwokwu shifts from teaching tools to helping you land a job. If you have been following this series, this is where the technical knowledge meets the job market.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1)
Part 1 covered GPU architecture and the gentle on-ramps (Thrust, OpenMP offload). Part 2 is where Morley shows what writing kernels actually looks like: CUDA for NVIDIA, then SYCL when you want one codebase targeting multiple devices.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Data Processing with Apache Spark
The last chapter of the main book tackles a problem the earlier NiFi chapters do not: what happens when the data source is a Raspberry Pi, a factory sensor, or some other small device that cannot run full NiFi? Crickard’s answer is Apache MiNiFi. Lightweight agent on the edge, full NiFi back at headquarters.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
Azure gets the same treatment AWS got in the previous chapter. Map Airflow components to managed services, survey the provider package, then build the movie-ranking pipeline again. Same goal, different cloud.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1)
CPUs are general-purpose workhorses. Sometimes you need a different animal. Chapter 14 introduces GPUs and other accelerators. This first post covers the architecture basics and the “easy mode” paths: library abstractions and OpenMP offload. Part 2 gets into raw CUDA and SYCL.
Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412, Wiley 2025)
Previous: Cloud Data Engineering Basics
The second half of Chapter 12 answers two questions every data engineer eventually faces: how much of the stack do you want to manage yourself, and how do you keep the cloud bill from giving finance a heart attack?
Data Science: The Hard Parts by Daniel Vaughan (ISBN 978-1-098-14647-4)
← Previous: From Prediction to Decisions | Next: A/B Tests and Experimentation
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Streaming Data with Apache Kafka
Chapter 13 showed you how to move streaming data. Chapter 14 asks what you do when the volume is too big for a single Python script or a NiFi flow. Crickard’s answer is Apache Spark. Download it, stand up a small cluster on your laptop, and learn PySpark DataFrames.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
Chapter 16 is where the cloud chapters get concrete. Harenslak and deRuiter map every Airflow component to an AWS service, then build a working movie-ranking pipeline with S3, Glue, and Athena.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412, Wiley 2025)
Previous: Big Data, Spark, and Hadoop
Chapter 12 opens with a history lesson that actually helps. Nwokwu walks from mainframes to on-prem servers to AWS in 2006, when cloud computing became real for most developers. The shift matters for data engineering because modern pipelines almost always run on someone else’s hardware.
Data Science: The Hard Parts by Daniel Vaughan (ISBN 978-1-098-14647-4)
← Previous: Storytelling in Machine Learning | Next: Incrementality and Causal Inference
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1)
At some point, one CPU with a handful of threads stops being enough. Chapter 13 is Morley’s tour of what changes when you scale up: more cores, more processes, more machines. The rubber duckies app fit on a laptop. Real workloads often do not.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Previous: Building a Kafka Cluster
Chapter 12 got Kafka running on your machine. Chapter 13 is where it starts to click. Crickard explains what Kafka actually is under the hood, wires it into the NiFi pipelines you built earlier, and shows you how to produce and consume messages with Python.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
Chapter 15 is the bridge chapter. You spent the last few chapters building DAGs, custom operators, and a real project. Now the book asks a harder question: where does all of this actually run?
Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412, Wiley 2025)
Chapter 11 is where the book stops talking about “a lot of data” in the abstract and starts explaining how companies like Netflix and Google actually process it. Nwokwu’s argument is simple: once your data outgrows a single machine, you need distributed systems. Full stop.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1)
Chapter 12 is the payoff moment. After chapters of file readers, regex parsing, and k-means clustering, Morley finally wires everything into one runnable program and then sits back to ask: what did we actually learn?
Data Science: The Hard Parts by Daniel Vaughan (ISBN 978-1-098-14647-4)
← Previous: Productionizing ML Models | Next: From Prediction to Decisions
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Section 3 of the book shifts from batch to streaming. Chapter 12 is infrastructure first: build a real Kafka cluster, not a single-broker tutorial that leaves you guessing how production works.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Data governance sounds boring until you realize it is the reason you do not get fined millions or repeat your life story to every department that calls. Chapter 10 ties together security and quality from the previous chapters into one framework.
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 11 Part 2 is where Morley ships the clustering code. Theory from Part 1 becomes KMeans, compute_score, best_kmeans_cluster, and compute_clusters. Tests catch real bugs, including a longitude flip at the north pole that only shows up on round-trip coordinate conversion.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Securing Apache Airflow | Next: Airflow in the Clouds
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Chapter 11 ended with a line that sticks: in ML, the scoring stage reigns. Chapter 12 is the follow-through. Vaughan gives data scientists a helicopter view of what “production ready” actually means, because at many companies, there is no dedicated MLOps team. You built the model. You ship it too.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Chapter 11 is the capstone for the production section. Crickard takes everything from Chapters 7 through 10 and builds a real pipeline: read JSON files from a data lake, load staging, validate, load warehouse, version every piece, deploy to production with variables.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
If you have trained enough real models, you have probably hit data leakage. Vaughan thinks it should rank higher than the “top 10 problems” lists give it credit for. Chapter 11 is entirely about this: what leakage looks like, how to spot it, and how to build habits that prevent it.
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 9 opens with the 2016 Uber breach. Hackers found API keys in a private GitHub repo, accessed AWS, stole 57 million user records, and Uber paid $100,000 to cover it up. Weak access controls, no real-time monitoring, credentials sitting in plain sight. That story sets the tone for everything that follows.
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 11 is the computational core of the rubber duckies project. All those file readers were building toward one question: where on Earth are these sightings clustering? Morley implements k-means from scratch. Choosing k and embedding geographic data correctly are the hard parts, not the iteration loop itself.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Operating Airflow in Production (Part 2) | Next: NYC Fastest Route Airflow Project
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
Software teams have dev, test, and production environments. Data engineering should work the same way. You build on one machine, prove it works somewhere that looks like prod, then promote it. Chapter 10 is about making that promotion less painful in NiFi.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 8 is a wake-up call. You can build the fanciest warehouse and the cleanest pipelines, but if the data itself is garbage, none of it matters. Nwokwu makes the case that data quality is not just a technical checkbox. It is a business strategy.
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 10 finishes the ingestion story. JSON and CSV were structured. Email reports are not. Morley builds a FreeTextReader using compile-time regular expressions from the ctre library to pull dates and coordinates out of unstructured .txt files.
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Linear regression is the first algorithm most of us learn and the first one we abandon. Gradient boosting eats its lunch on predictive accuracy. Fair enough. But Vaughan’s Chapter 10 makes a case that skipping OLS intuition leaves you blind to problems that hit random forests and boosting just as hard.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Operating Airflow in Production (Part 1) | Next: Securing Apache Airflow
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
You built a pipeline that is idempotent, version controlled, and ready for production. Great. But production means things break when you are not looking. Elasticsearch goes down. A network hiccup kills a connection. Bad data slips through. Chapter 9 is about knowing when that happens without staring at a screen all day.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
The second half of Chapter 7 answers a question every data team hits eventually: how do you run live dashboards AND historical reports without building two completely separate systems? Then it gets practical with orchestration and a hands-on Airflow lab.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Airflow Best Practices (Part 2) | Next: Operating Airflow in Production (Part 2)
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 9 tackles the first two file formats in the rubber duckies pipeline. CSV and JSON are solved problems with good libraries. Morley’s actual challenge is designing one interface that hides how each format gets parsed.
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Most data science courses teach you on real datasets. Daniel Vaughan spends Chapter 9 asking a different question: what if you built the data yourself? Not synthetic data from a generative model. A hand-written data generating process (DGP) where you control every assumption. That shift sounds academic until your gradient boosting model looks brilliant on paper and falls apart in production.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
You can build a beautiful NiFi pipeline and still be one bad drag-and-drop away from losing hours of work. Chapter 8 fixes that gap. Version control for data pipelines is not optional in production. Crickard treats data engineers like software engineers here, which is the right framing. You would not write application code without Git. You should not run production flows without versioning either.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Airflow Best Practices (Part 1) | Next: Operating Airflow in Production (Part 1)
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 7 is where the book stops talking about storage and starts talking about movement. Data pipelines are the plumbing. They collect, ingest, process, store, and serve data. Sometimes on a schedule, sometimes the second an event happens.
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 8 is where the rubber duckies project gets a real executable named duckies. Morley builds a minimal but extensible CLI around Boost program_options, spdlog, and OpenMP. Even if you are the only user, he argues, --help pays for itself the first time you forget what flags exist.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly, 2024
Previous: Narratives and Storytelling in Data Science | Next: Simulation and Bootstrapping
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Section 2 of the book starts here. The vibe shifts from “learn the tools” to “do not break production.” Chapter 7 covers three ideas that separate hobby pipelines from ones you can run at 3 a.m. without panic: staging, validation, idempotency, and atomicity. Crickard uses a fake “Widget Co” sales example and a people-data pipeline he actually runs, though he admits that one is low-stakes and missing pieces a critical system would need.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Running Airflow Tasks in Containers (Part 2) | Next: Airflow Best Practices (Part 2)
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly, 2024
Previous: What’s in a Lift? | Next: Datavis: Choosing the Right Plot
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Part 1 was theory. Chapter 7 kicks off Part 2 with a full project. Morley hands you a client brief, sample data, and then does the thing most tutorials skip: he thinks before he codes.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 6 is where the book stops feeling like a tutorial and starts feeling like a real project. Crickard builds a pipeline that pulls citizen service requests from SeeClickFix (think 311-style complaints: graffiti, potholes, abandoned cars, needles, people ignoring social distancing during COVID). The data lands in Elasticsearch. Then you build a Kibana dashboard on top. He says he still runs this pipeline every 8 hours. That detail matters. This is not a throwaway demo.
Chapter 6 of Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412) is a big one. It moves beyond traditional databases into the storage systems that power modern analytics. The fictional bakery Dough & Delight runs through every example, which keeps things grounded.
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Running Airflow Tasks in Containers (Part 1) | Next: Airflow Best Practices (Part 1)
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 6 is where Morley stops talking about solving one-off problems and starts talking about what happens when other people (including future you) have to use your code. The whole point is simple: code only matters if someone can actually reuse it.
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly, 2024
Previous: Building Business Cases for Data Science Projects | Next: Narratives and Storytelling in Data Science
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapters 3 and 4 taught you how to move data in and out of files and databases. Chapter 5 asks the next obvious question: what do you do with the data in the middle? Often the answer is clean it, reshape it, and sometimes bolt on extra context from another source. Crickard uses real Albuquerque e-scooter trip data for all of this, which makes the examples feel grounded instead of toy-like.
The second half of Chapter 5 in Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412) answers the question every data engineer eventually faces: your schema is clean, but your queries are slow. Now what?
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly, 2024
Previous: 2×2 Designs to Simplify Data Science | Next: What’s in a Lift?
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
The back half of Chapter 5 is where Morley admits vectors are not always enough. Linked structures, ordered maps, and hash tables each trade something away. He also runs real benchmarks so you can see the gap instead of guessing.
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Testing Airflow DAGs (Part 2) | Next: Running Airflow Tasks in Containers (Part 2)
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 4 is where the book stops playing with files on disk and starts working with databases. For most data engineers, this is the daily bread.
Chapter 5 of Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412) shifts from writing queries to designing the databases those queries run against. And honestly, this might be the most important chapter for long-term data engineering work.
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly Media (2024)
Previous: Growth Decompositions and Tailwinds | Next: Building Business Cases for Data Science
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 5 is the payoff for Chapters 3 and 4. You know how algorithms scale and how CPUs read memory. Now Morley asks: what container actually holds your data, and does its layout help or hurt?
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Testing Airflow DAGs (Part 1) | Next: Running Airflow Tasks in Containers (Part 1)
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 3 is where the book starts feeling like data engineering instead of systems administration.
The second half of Chapter 4 in Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412) is where SQL goes from “filter a table” to “actually analyze data across tables.” This is the part that makes SQL feel like a real data engineering skill.
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly Media (2024)
Previous: Metrics Design for Data Scientists | Next: 2×2 Designs for Data Science
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Building Custom Airflow Components (Part 2) | Next: Testing Airflow DAGs (Part 2)
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Part 2 of Chapter 4 is where Morley gets hands-on with SIMD, shows how branch style changes generated code, and reminds you that the operating system sits between your pointers and physical RAM.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 told you what data engineering is. Chapter 2 asks you to build the whole workshop on one Linux machine.
Data Engineering for Beginners by Chisom Nwokwu (ISBN 9781394325412) picks up right where the database fundamentals chapter left off. Chapter 4 is where SQL stops being abstract and starts feeling like a real tool you will use every day as a data engineer.
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Building Custom Airflow Components (Part 1) | Next: Testing Airflow DAGs (Part 1)
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 3 zooms in on storage. After the life cycle overview, Nwokwu focuses on databases, the tool data engineers spend a huge amount of time with. This chapter is dense but practical. It covers what databases are, how relational and NoSQL systems differ, and how to pick the right one for your project.
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly Media (2024)
Previous: Creating Value with Data Science: The “So What” Framework | Next: Growth Decompositions and Tailwinds
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 4 answers a question Chapter 3 raised: your algorithm might be correct on paper and still slow because the CPU spent half its time waiting on RAM. Morley gives a compressed tour of modern hardware so C++ programmers know what they are actually driving.
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 of Paul Crickard’s book is the orientation lecture. No installs yet. No code. Just a clear picture of what data engineering is, who does it, and what tools show up again and again.
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
The second half of Chapter 3 is a tour of algorithm design styles, then a hard turn toward what the hardware actually rewards. Morley’s message throughout: reuse known solutions first, then get clever only when you have to.
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Airflow and External Systems | Next: Building Custom Airflow Components (Part 2)
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly Media (2024)
Previous: Series Intro | Next: Metrics Design for Data Scientists
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 2 is where Data Engineering for Beginners stops talking about data in general and starts defining the job. Nwokwu answers the question everyone asks: what does a data engineer actually do all day?
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I’m starting a walkthrough of Data Engineering with Python by Paul Crickard. This is not a copy-paste summary. It’s my retelling of each chapter with notes on what stuck, what felt dated, and what still holds up.
Book: The C++ Programmer’s Mindset
Author: Sam Morley
ISBN: 978-1-83588-842-1
Chapter 3 is where Morley stops talking about what code looks like and starts talking about how fast it runs. Algorithms are step-by-step instructions that bridge a problem and an implementation. They are language-agnostic, which is why every serious programmer should know how to read, analyze, and write them.
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Triggering Airflow Workflows | Next: Building Custom Airflow Components (Part 1)
Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly Media (2024)
I’m starting a walkthrough of Daniel Vaughan’s Data Science: The Hard Parts. Over the next few weeks I’ll post one chapter at a time, retelling what the book says and adding my own take.
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 1 of Data Engineering for Beginners opens with a line you have probably heard before: data is the new oil. Nwokwu does not just repeat the slogan. She breaks down what data actually means, where it came from, and why the type of data you are working with changes everything downstream.
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1, Packt 2025)
Part 1 of Chapter 2 covered problem categories, standard algorithms, and functions as interfaces. This post finishes the chapter: functions inside algorithms, function objects, classes, templates, concepts, and traits.
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
I picked up Data Engineering for Beginners because I kept hearing about data engineering at work without fully understanding what the job actually looked like day to day. Analysts talk about dashboards. ML folks talk about models. But who builds the pipes that get clean data to them? That gap is exactly what this book tries to fill.
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Airflow Branching and the TaskFlow API | Next: Airflow and External Systems
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1, Packt 2025)
Chapter 1 taught computational thinking in the abstract. Chapter 2, “Abstraction in Detail,” asks a sharper question: how do C++ language features help you find the right abstractions in real problems?
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
The first half of chapter 5 was DAG shape and branching. Sections 5.4 through 5.6 answer: when exactly does a task run, how do tasks pass small bits of state, and can Python-heavy DAGs look like normal Python again?
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 5 is where DAGs stop looking like straight lines. The book covers dependency shapes, branching when your data source changes, and skipping tasks when a condition is not met. This post covers sections 5.1 through 5.3. Trigger rules, XCom, and Taskflow API are the next post.
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1, Packt 2025)
Part 1 of Chapter 1 covered decomposition, abstraction, and patterns. This post finishes the chapter: what algorithms actually are, how to read them, and how modern C++ plus good engineering habits help you iterate faster on solutions.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 3 showed {{ds}} in a curl command. Chapter 4 explains why that works, what else you can template, and how operators talk to Postgres without you writing connection boilerplate.
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1, Packt 2025)
Chapter 1 of Morley’s book is called “Thinking Computationally,” and it sets the tone for everything that follows. This is not a chapter about std::vector tricks. It is about how programmers break down problems, find structure in messy data, and eventually write code that fits.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 2 ended with @daily and a glimpse of the tree view. Chapter 3 goes deep on scheduling: how Airflow slices time, how to process only today’s data instead of everything every day, and what makes a task safe to rerun.
The C++ Programmer’s Mindset by Sam Morley (ISBN 978-1-83588-842-1, Packt 2025)
Most C++ books teach you syntax, containers, and maybe a few design patterns. Sam Morley’s book takes a different angle. It asks: how do you actually think when you sit down with a hard problem? The answer is not “memorize more STL headers.” It is computational thinking, paired with the specific tools C++ gives you to turn that thinking into working code.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 1 was theory. Chapter 2 is hands-on. The book follows John, a rocket enthusiast who wants upcoming launch images on his machine. The Launch Library API returns JSON with image URLs. John’s plan: download metadata, fetch pictures, notify when done. That becomes your first real Airflow DAG.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
I picked up Data Pipelines with Apache Airflow because I kept hearing the same complaint from data teams: the volume is fine, but the coordination is a mess. Megabytes turned into gigabytes per minute, and nobody had a clean way to run the same steps in order, on a schedule, with visibility when something broke. Chapter 1 is the orientation chapter. No install yet. Just: what is a pipeline, why draw it as a graph, and where Airflow sits in the zoo of workflow tools.
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
I’m working through Data Pipelines with Apache Airflow and writing up each chapter as a blog post. Not a dry summary. More like a retelling with my own reactions, the parts that clicked, and the parts that made me pause.
We made it. Twenty posts. Fourteen chapters. One very thorough book about building data pipelines on Amazon Web Services.
When I started this retelling series back in December, I said the goal was to walk you through “Data Engineering with AWS” by Gareth Eagar in plain language, one chapter at a time. No jargon walls. No assumed knowledge. Just the concepts, the tools, and how they fit together. I hope I delivered on that.
So we made it through all 11 chapters of Ravi Mishra’s HashiCorp Infrastructure Automation Certification Guide. Here’s my honest take on the whole thing.
And that’s a wrap. We made it through all 10 chapters of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi. Here’s what I think after spending time with this book.
Chapter 11 is the glossary chapter. In most books, glossaries sit at the back collecting dust. But honestly, if you’re studying for the Terraform certification, this is the chapter you’ll keep coming back to. Ravi Mishra collects every term that showed up throughout the book and defines them in one place.
Welcome back. In Part 1 we went through garbage collection algorithms, reference counting, mark-and-sweep, and generational collection. Now let’s finish up Chapter 10 of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi. This second half covers cache management, space allocation on the stack, pointers, memory management tips, and profiling.
This is post 20 in my Data Engineering with AWS retelling series.
We made it. Thirteen chapters of pipelines, transforms, orchestration, security, querying, visualization, and machine learning. Chapter 14 is the final chapter. It does not introduce a new AWS service. Instead, it zooms out. Way out. It looks at how data engineering works in the real world, shows you case studies from Spotify and Netflix, and points you toward where the field is heading next.
Up to this point in the book, everything has been about running Terraform on your own machine. Local state files, local commands, you doing everything manually. That works fine when it’s just you. But the moment your team grows beyond a couple of people, things get messy fast. Chapter 10 introduces the solutions HashiCorp built for exactly that problem: Terraform Cloud and Terraform Enterprise.
We’ve spent nine chapters building data structures, stacking nodes, balancing trees, linking lists. But here’s a question we haven’t really asked yet: what happens to all that memory when we’re done with it?
Chapter 9 is where the pieces from earlier chapters finally click together. You learned about modules. You learned about configuration files. Now you take those modules, combine them, and deploy real infrastructure at scale. That’s what Terraform stacks are about.
Welcome back. In Part 1 we covered graph fundamentals, social networks, and map layouts. Now Kommadi finishes Chapter 9 with two more practical topics: knowledge graphs and sparse matrix representation. Both build on the graph foundation from Part 1, but they’re different enough to deserve their own discussion.
Chapter 8 of Ravi Mishra’s book is one of those chapters that sounds basic on the surface but actually ties a lot of loose ends together. You’ve been writing Terraform code for seven chapters now, but this is where you stop and really understand the anatomy of a configuration file. What goes where, why it matters, and how the same patterns work across GCP, AWS, and Azure.
Chapter 9 of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi shifts gears into graphs and network structures. If you’ve been following along, we spent the last few chapters on searching, sorting, and hashing. Now we’re getting into something that models the real world more directly: connections between things.
Part 1 of this chapter covered the theory: what modules are, where they come from, meta-arguments, and source types. Now it’s time to actually build stuff. Part 2 is where Ravi Mishra walks you through creating real Terraform modules for all three major cloud providers: Azure, AWS, and GCP.
Welcome back. In Part 1 we covered the sorting side of Chapter 8, from bubble sort all the way to quick sort. Now we’re picking up the second half: searching algorithms, recursion, and hashing. These are the tools you use when you already have your data and need to find stuff in it, or when you need to transform it for fast lookups.
Chapter 7 is where the book gets into one of the most practical parts of Terraform: modules. If you’ve been writing Terraform configs so far and copy-pasting blocks between projects, this chapter is basically the answer to “there has to be a better way.”
Chapter 8 of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi is called “Classic Algorithms.” It covers sorting, searching, recursion, and hashing. That’s a lot of ground, so we’re splitting it into two parts. This first part is all about sorting.
Chapter 6 is where the book stops talking about individual commands and starts showing you how they all fit together. If the previous chapters were about learning the notes, this one is about playing the song.
Welcome back. In Part 1 we went through dictionaries and TreeSets. This second half of Chapter 7 wraps up TreeSets with synchronized and mutable variants, then moves into some cool mathematical sequences implemented in Go. We also talk about common anti-patterns the book warns about when working with these data structures.
Chapter 5 is where the book gets hands-on with the Terraform CLI. Up to this point you’ve been learning concepts. Now it’s time to learn what you actually type into the terminal.
We’re into Chapter 7 of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi, and this is where things get interesting. The chapter is about dynamic data structures, which are basically collections that can grow and shrink as needed. No fixed sizes, no guessing how much memory you need upfront.
This is post 19 in my Data Engineering with AWS retelling series.
Throughout this book, we have been ingesting data, transforming data, storing data, querying data, and visualizing data. All of that is incredibly useful on its own. But there is a whole other level where data gets really powerful: when you use it to teach a machine to make predictions.
The second half of Chapter 4 in Ravi Mishra’s Terraform certification guide covers three things that will make your Terraform code way less repetitive and way more debuggable: loops, built-in functions, and debugging tools.
Welcome back to Chapter 6 of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi. In Part 1 we covered singly and doubly linked lists. Now we pick up where we left off: circular linked lists, ordered lists, and unordered lists.
Chapter 4 is where the book gets serious. We’re past the basics of providers, resources, and variables. Now we’re talking about where Terraform stores its memory (backends) and how you can run scripts as part of your infrastructure deployments (provisioners).
Chapter 6 of “Learn Data Structures and Algorithms with Golang” is all about heterogeneous data structures. That’s a fancy way of saying “data structures that can hold different types of data.” Think integers, floats, strings, whatever you need, all mixed together. Linked lists and ordered lists are the main examples here.
In Part 1 we covered providers, resources, and variables. Now let’s talk about the other two pieces of the puzzle: outputs and data sources. These are the things that make your Terraform configs actually talk to each other and to existing infrastructure.
Welcome back to Part 2 of Chapter 5 from “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi. In Part 1 we looked at arrays and multi-dimensional arrays. Now we get into the fun stuff: matrix operations, special matrix types, and tensors.
Chapter 3 of Ravi Mishra’s certification guide is where Terraform stops being theoretical and starts getting real. You installed it in Chapter 2, now you actually write configuration code. This first half covers three fundamentals: providers, resources, and input variables across Azure, AWS, and GCP.
Chapter 5 of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi shifts gears from trees and hash tables into something more math-heavy: homogeneous data structures. That basically means data structures where every element is the same type. Think arrays of integers, matrices of floats, that kind of thing.
Chapter 2 of the book is the “roll up your sleeves” chapter. No more theory about what IaC is or why Terraform exists. Now you actually install the thing and get it running on your machine.
Welcome back. In Part 1 we covered binary search trees and AVL trees. Now we’re finishing up Chapter 4 of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi. This second half covers B-trees, B+ trees, tables, containers, circular linked lists, and hash functions.
Chapter 1 of Ravi Mishra’s book sets the stage for everything that follows. Before you touch a single Terraform command, you need to understand what Infrastructure as Code actually is and why it exists. Let’s break it down.
Up to this point in the book, everything we covered was linear. Lists, stacks, queues, heaps, all of them store data in a straight line. One element after another. Chapter 4 is where things get interesting because we’re moving into non-linear data structures.
So I picked up this book called HashiCorp Infrastructure Automation Certification Guide by Ravi Mishra. It’s published by Packt (ISBN: 978-1-80056-597-5) and it’s basically a full walkthrough of Terraform, from zero to certification-ready.
Welcome back. In the first half of Chapter 3, we covered linked lists, sets, and tuples. Now we’re finishing the chapter with queues, stacks, and heaps. These three come up constantly in real software, from job schedulers to undo buttons to sorting algorithms. Let’s get into it.
This is post 18 in my Data Engineering with AWS retelling series.
We have spent eleven chapters ingesting data, transforming data, cataloging data, querying data. But here is a simple truth: nobody wants to stare at 10,000 rows in a spreadsheet. Our brains are not built for that. We process pictures way faster than text. A well-designed chart can tell you in two seconds what would take twenty minutes to figure out from raw numbers.
Chapter 3 is where the book starts getting into the real stuff. We’re past the basics of Go syntax and data types, and now it’s time to build actual data structures from scratch. This first half covers three linear data structures: linked lists (both single and double), sets, and tuples.
Welcome back. In Part 1 we covered arrays, basic slices, two-dimensional slices, and maps. That was the foundation. Now Kommadi moves into the more interesting Go patterns: variadic functions, defer and panic, and a full CRUD web application that ties it all together. He also shows more advanced slice operations along the way.
Welcome to Chapter 2 of our walkthrough of “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi. This is Part 1, where we cover the Go-specific building blocks you need before getting into the heavier data structure topics. Think of this as your toolkit chapter.
In Part 1 we covered what data structures are, how they get classified, and a bunch of structural design patterns. Now we’re getting to the fun stuff: algorithms, how to measure their performance, and a few classic algorithm types.
Chapter 1 of the book jumps right into classifying data structures and showing some structural design patterns in Go. There’s a lot packed in here, so I’m splitting it into two parts. This first part covers what data structures are, how they’re classified, and a bunch of design patterns with Go code.
So I picked up this book called “Learn Data Structures and Algorithms with Golang” by Bhagvan Kommadi (Packt Publishing, 2019, ISBN: 978-1-78961-850-1). And I’m going to retell the whole thing here on the blog, chapter by chapter.
This is post 17 in my Data Engineering with AWS retelling series.
You have a data lake. Terabytes of files sitting in S3 across landing zones, clean zones, and transform zones. The data is there. But how do you actually ask it questions? You could spin up a database, load everything into it, and then query. But that defeats the purpose of having a data lake in the first place.
This is post 16 in my Data Engineering with AWS retelling series.
Up to this point in the book, we have been doing everything by hand. Click this button in the console. Run that Glue job manually. Trigger a crawler. Upload a file. It works fine for learning. But imagine doing that in production, every day, at 3 AM, across dozens of data sources. No thanks.
Previous: What’s Next for Blockchain? The Research Frontier
And that’s a wrap! We’ve spent the last 12 days diving deep into Blockchain and Banking by Pierluigi Martino. This book really changed how I think about the money in my pocket (and on my screen).
Previous: The Future of Banking: My Final Take
We’ve covered a lot of ground this week. But as Pierluigi Martino points out in the final chapter of Blockchain and Banking, we’re still just at the beginning. Most of what we “know” about blockchain in banking is still just theory.
We have reached the end of our deep dive into Big Data on Kubernetes by Neylson Crepalde. It has been a massive journey, moving from basic Docker containers to complex, real-time AI pipelines.
We have built some incredible pipelines over the last few posts. But if you were to take what we’ve built and put it into production today, you’d quickly realize that there is a lot more to managing a platform than just getting the YAML files right.
Previous: Smart Contracts in Real Courts: A Legal Nightmare?
We’ve spent the last week looking at the tech, the business side, and the rules of blockchain. Now, we’re at the end of Pierluigi Martino’s book, Blockchain and Banking.
In the last post, we saw how to give an AI model a “memory” using RAG. But the real game-changer in the Generative AI world is when you let the model actually do things.
Previous: The Rules of the Road: Regulating Blockchain
Yesterday we looked at how governments are trying to regulate the “wild west” of crypto. Today, we’re looking at something even trickier: Smart Contracts.
We have spent this whole series talking about “Big Data”—Spark, Kafka, and SQL engines. But the hottest topic in tech right now isn’t just data processing; it’s Generative AI.
Previous: Rebuilding the Bank: New Business Models
Most people think of blockchain as a “lawless” digital frontier. And for a while, it kind of was. But in Chapter 5, Pierluigi Martino explains how governments around the world are finally catching up and making some rules.
Batch processing is great for historical reports, but what if you need to know what’s happening right now? In the final part of Chapter 10, Neylson Crepalde shows us how to build a world-class Real-Time Pipeline on Kubernetes.
Previous: The Threats Banks Are Actually Scared Of
If you think banks are just going to use blockchain to do the “same old thing” a little bit faster, think again. In Chapter 4, Pierluigi Martino uses a popular tool called the Business Model Canvas to show how blockchain is actually rebuilding the entire structure of a bank.
In our last post, we checked the infrastructure. Now, let’s build the actual pipeline. Neylson Crepalde uses the IMDB dataset to demonstrate a professional batch workflow.
Previous: Banks vs. Blockchain: Friends or Foes?
Yesterday, we talked about how banks are using blockchain to get faster. But today, we’re looking at why they’re still sweating. Pierluigi Martino points out that while the tech is a great tool, it’s also a massive threat to their bottom line.
Previous: Beyond Crypto: The Real Power of Smart Contracts
The relationship between big banks and blockchain is… complicated. It’s like that “frenemy” relationship you had in high school. In Chapter 3, Pierluigi Martino dives into why the banking world is both terrified and obsessed with this tech.
We have spent the last few weeks looking at individual tools like Spark, Airflow, and Kafka. But in the real world, these tools don’t live in isolation. They need to talk to each other to form a complete data pipeline.
This is post 15 in my Data Engineering with AWS retelling series.
In Part 1, we looked at Redshift internals – clusters, slices, distribution styles, sort keys. All the pieces that make a data warehouse fast. But a warehouse sitting in isolation is not very useful. Data needs to flow in from your data lake, and sometimes it needs to flow back out. Part 2 of Chapter 9 covers that bridge between S3 and Redshift, including Redshift Spectrum, the COPY and UNLOAD commands, and a hands-on exercise that ties it all together.
Previous: Blockchain 101: More Than Just Bitcoin
Yesterday we talked about what blockchain is. Today, we’re looking at what it actually does. Most people stop at Bitcoin, but Pierluigi Martino shows us that the real magic happens when we move beyond simple payments.
Trino is great for querying your historical data on S3, but for real-time streams and text-heavy search, you need something different. In the second half of Chapter 9, Neylson Crepalde introduces the industry standard for real-time analytics: Elasticsearch and Kibana.
Previous: How Fintech Started the Revolution
If you think blockchain is just another word for Bitcoin, this post is for you. In the second chapter of his book, Pierluigi Martino peels back the curtain on what this tech actually is.
You’ve built your ingestion, you’ve processed your data with Spark, and it’s all sitting neatly in your S3 “Gold” bucket. Now what? You can’t ask every business analyst to learn PySpark just to see last month’s sales.
In my last post, we got Spark running natively on Kubernetes. Now, it’s time to bring in the conductor (Airflow) and the nervous system (Kafka). This is where your cluster starts to feel like a real data platform.
Previous: Why This Tech is Actually a Big Deal
Before we talk about blockchain, we need to talk about its big brother: fintech. In the first chapter of Blockchain and Banking, Pierluigi Martino takes us through how we got here.
Ever feel like the banking world is just a bunch of old buildings and slow apps? Well, things are actually moving pretty fast behind the scenes. I just finished reading Blockchain and Banking: How Technological Innovations Are Shaping the Banking Industry by Pierluigi Martino, and it’s a real eye-opener.
We’ve explored Spark, Airflow, and Kafka as individual tools. But the real goal of Neylson Crepalde’s book is to show you how to run them all as a cohesive “stack” on Kubernetes. In Chapter 8, we finally start the heavy lifting of deployment.
Architecture is great, but let’s actually run some code. In the second half of Chapter 7, Neylson Crepalde walks us through setting up a multi-node Kafka cluster right on our local machine using Docker Compose.
In the world of big data, “batch” is no longer enough. We need data the second it happens. Whether it’s tracking stock prices, monitoring website traffic, or detecting fraud, you need a system that can handle massive streams of events with zero downtime.
This is post 14 in my Data Engineering with AWS retelling series.
Your data lake is humming along. Data lands in S3, gets cleaned, transformed, cataloged. Athena lets you run SQL queries on it. So why would you need anything else?
In the last post, we got Airflow running. Now, let’s talk about how to actually use it. The heart of Airflow is the DAG—the Directed Acyclic Graph.
If Spark is the engine, then Apache Airflow is the conductor. In a modern data stack, you rarely have just one job running in isolation. You have ingestion, cleaning, processing, and delivery—and they all have to happen in a specific order.
In the last post, we looked at Spark’s architecture. Now, let’s talk about how you actually write code for it. Neylson Crepalde highlights two main ways to interact with Spark: the DataFrame API and Spark SQL.
If there is one tool that defined the “Big Data” era, it’s Apache Spark. It’s the engine that handles everything from terabyte-scale ETL to complex machine learning. In Chapter 5, Neylson Crepalde breaks down exactly how Spark works and why it’s so powerful on Kubernetes.
We’ve talked about the architecture, but what about the actual tools? To build a modern data lakehouse on Kubernetes, you need a specific set of tools that can handle scale, automation, and speed.
We’ve all heard the terms “Data Warehouse” and “Data Lake,” but do you actually know why we keep switching between them? In Chapter 4 of Big Data on Kubernetes, Neylson Crepalde gives a masterclass on how data architecture has evolved to keep up with the modern world.
Testing things locally with Kind is great, but big data usually needs big iron. In this part of the hands-on journey, Neylson Crepalde shows us how to scale up to a managed cloud environment.
This is post 13 in my Data Engineering with AWS retelling series.
We have spent the last several chapters ingesting data, transforming data, optimizing data. Pipelines everywhere. But here is the question nobody asks often enough: who is actually going to use all of this?
Reading about architecture is one thing, but actually seeing a cluster run is where it sticks. In the third chapter of Big Data on Kubernetes, Neylson Crepalde moves from theory to practice.
In the last post, we talked about the “brain and muscles” of a Kubernetes cluster. But how do we actually tell that brain what to do? We use Objects.
If you want to run big data workloads on Kubernetes, you have to understand how the system is actually put together. It’s not just “magic magic cloud stuff”—it’s a carefully coordinated cluster of machines.
In my last post, we talked about why containers are the bedrock of modern data engineering. But honestly, just running other people’s images only gets you so far. The real magic happens when you start packaging your own custom code.
If you are working with data today, you can’t really ignore containers. They have become the standardized unit for how we develop, ship, and deploy software. But why do we care so much about them in the big data world?
We are living in a world where data is basically everywhere. From your phone to social media and every single online purchase, the amount of info we generate is staggering. But here’s the thing: just having data isn’t enough. You have to be able to process it, and that’s where things get complicated.
This is post 12 in my Data Engineering with AWS retelling series.
In Part 1, we covered the generic data preparation transforms: converting to Parquet, partitioning, PII protection, and data cleansing. Those transforms work on individual datasets and do not need much business context. Now we get to the transforms that actually create business value. The ones that combine multiple datasets, add context, flatten structures, and produce the tables that analysts and dashboards consume.
And that’s a wrap. Over the past 24 days, we walked through every chapter of Laura Funderburk’s Building Natural Language and LLM Pipelines. Here are my final thoughts on the book as a whole.
And that’s it. Eighteen posts. Thirteen chapters. One complete walkthrough of “Data Engineering for Beginners” by Chisom Nwokwu.
When I started this series, I said I wanted to retell the book in my own words. Not a summary, not a copy. My take on what each chapter covers and why it matters. Now that I’m at the end, let me step back and share my overall impressions.
Twenty-two posts later, we are done. This was my retelling of “Data Engineering with Google Cloud Platform” by Adi Wijaya (2nd edition, Packt Publishing, 2024, ISBN 978-1-83508-011-5). Time to look back, share what stuck with me, and give an honest assessment.
That’s it. Fifteen chapters, seventeen posts, and one complete walkthrough of Paul Crickard’s Data Engineering with Python (Packt, 2020, ISBN: 978-1-83921-418-9).
Nineteen posts. Sixteen chapters. One book. And here we are at the end.
When I started this retelling of Data Science Foundations: Navigating Digital Insight by Stephen Mariadas and Ian Huke (ISBN: 978-1-78017-6994, BCS 2025), I was not sure how it would go. Some books lose steam halfway. Some start strong and fizzle. But this one stayed consistent from first chapter to last.
So we made it through the whole book. And honestly? It was worth the ride.
The biggest thing Scavetta and Angelov got right is the framing. They didn’t write a “Python is better” or “R is better” book. They wrote a “both are useful, here’s when to use which” book. And that’s the mature take.
This is the last technical chapter of the book. Everything before this was about skills, tools, and concepts. Chapter 13 is about what you do with all of that knowledge. How you actually get a job in data engineering.
In Part 1 we went through the quiz questions, extra GCP services, and how the book ties everything together. Now let’s talk about the stuff that matters after you close the book: getting certified, planning your career, and figuring out what comes next.
So we made it. Chapter 16 is the conclusion of Data Science Foundations by Stephen Mariadas and Ian Huke. And like most good conclusions, it does not introduce anything new. Instead it steps back and asks: what did we learn, and where is all of this going?
The appendix of “Python and R for the Modern Data Scientist” is basically a bilingual dictionary. It runs about 40 tables long and covers everything from package management to indexing. You could spend a whole afternoon reading through it.
You have NiFi running. Kafka is streaming. Spark is processing. But what about the data source? What happens when your data comes from a tiny sensor or a Raspberry Pi that can barely run a web browser?
In the previous post, we looked at how agentic architectures evolved from brittle sequential chains (V1) through router patterns (V2) to resilient supervisors (V3). Now Funderburk puts those architectures under stress. The results are honestly a little scary.
The epilogue of Funderburk’s book is where everything clicks together. All the individual skills from earlier chapters (pipelines, RAG, tool contracts, Haystack components, LangGraph orchestration) get assembled into a single architectural argument. And that argument is surprisingly clear: separate the doing from the thinking.
Chapter 12 is the one where everything moves to the cloud. If you’ve been following along, we’ve been talking about databases, pipelines, data quality, security, governance, and big data. All of that can run on your own hardware. But most teams today don’t do that. They use cloud providers. This chapter explains why, and more importantly, how.
Chapter 13 is the last chapter in the book, and it’s different from everything that came before. No new GCP services, no hands-on exercises, no Terraform scripts. Instead, Adi Wijaya steps back and talks about the bigger picture: certifications, where data engineering is heading, and how to actually feel confident in this role.
You have streaming data. You have batch data. You have a lot of it. Now you need to actually process it. Fast. On more than one machine.
Theory is nice. But does any of it work in the real world? Chapter 15 of Data Science Foundations by Stephen Mariadas and Ian Huke answers that with five case studies. Real people, real problems. One of them technically failed. And that is part of the point.
The whole book has been building to this. Six chapters of philosophy, syntax comparisons, and interoperability tricks. Now Chapter 7 drops a real project on the table. Build it with both languages. Together. Start to finish.
At some point, your data gets too big for one machine. That’s not a hypothetical. Netflix, Google, Amazon, they all hit that wall years ago. The question is: what do you do when a single server can’t keep up?
In Part 1 we covered the theory behind CI/CD and ran through a basic Cloud Build exercise with a Python project. Unit tests ran automatically on every push. Broken code got caught before it reached production. Good stuff, but that was a simple calculator script. Now we need to connect this to real data engineering work.
Everyone has an opinion about AI. Your coworker worries robots will take his job. Your cousin swears ChatGPT wrote his college essay. Chapter 14 of Data Science Foundations by Stephen Mariadas and Ian Huke explains what machine learning and artificial intelligence actually are. And how data science connects to all of it.
In the first half of Chapter 9, Funderburk covered hardware limitations and the four big problems with LLMs. Now she gets to the good stuff: the protocols and frameworks that are actually solving those problems.
Up to this point in the book, data pipelines have been about moving data that already exists. Query a database, read a file, process it, store it. The data sits still and you go get it.
Chapter 6 is where the book finally delivers on its promise. All that talk about using both languages together? This is where it actually happens. Rick Scavetta walks through the nuts and bolts of making Python and R talk to each other in the same project.
Up to this point in the book, everything has been batch processing. You query a database, get a full dataset, transform it, load it somewhere. The data sits still while you work on it.
This is post 11 in my Data Engineering with AWS retelling series.
You have data sitting in your data lake. Raw CSV files, JSON dumps, database extracts. It is all there, technically available, but trying to run analytics on it is painfully slow and expensive. This chapter is about fixing that. Transforming raw data into something optimized, clean, and ready for actual use.
Chapter 12 is a shift from everything we have done so far. Until now, we were learning how to build things: pipelines, data lakes, warehouses, streaming systems. Now the question is: how do you ship all that stuff to production without breaking things? The answer is CI/CD.
Data governance sounds like something a committee of suits invented to make your life harder. But here’s the thing: without it, everything falls apart quietly.
You did the hard work. You collected data, cleaned it, tested your models. And now you need to tell someone what you found. This is where Chapter 13 of “Data Science Foundations” by Stephen Mariadas and Ian Huke comes in. Communication. The part that separates useful data science from data science that nobody cares about.
Chapter 9 of Laura Funderburk’s book takes a step back from building things and looks forward. What’s coming next for NLP and LLM systems? Where are the bottlenecks? What’s changing?
Chapter 5 is where Boyan Angelov gets practical about the question everyone dances around: which language should you actually use for which job?
You learned the individual tools. You learned the deployment strategies. Now Chapter 11 of Data Engineering with Python by Paul Crickard puts it all together. This is the chapter where you build a complete, production-grade data pipeline from start to finish.
This is where everything from Chapter 8 comes together. We’ve built NER pipelines, text classification tools, sentiment analyzers. Now Funderburk wires them into a multi-agent graph that can handle complex queries end to end.
Nobody ever got promoted for building the cheapest data pipeline. But plenty of people have gotten uncomfortable phone calls from their CFO after a runaway BigQuery bill. Chapter 11 is about the money side of GCP, and I think this is one of the most practical chapters in the book.
You built a model. It runs. It gives you numbers. But does it actually work? That is what Chapter 12 of “Data Science Foundations” by Stephen Mariadas and Ian Huke is about. Building a model is one thing. Trusting it is something else.
In 2016, hackers stole personal data of 57 million Uber users and drivers. How? Someone left API credentials in a private GitHub repo. The attackers grabbed those keys, got into AWS, and downloaded everything. Uber didn’t even notice for a year. When they finally found out, they paid the hackers $100,000 to delete the data and kept quiet about it.
Chapter 4 is where the book stops teaching you the languages and starts telling you when to use which one. This is Part III, “The Modern Context,” and Boyan Angelov takes the lead here. The question is simple: given a specific data format, which language gives you a better experience?
In Part 1 we covered how data governance breaks into three pillars (usability, security, accountability) and went through metadata, Dataplex search, access control in BigQuery, and the Sensitive Data Protection service for finding PII. Now let’s pick up where we left off: understanding what SDP actually finds, and then moving into the accountability pillar.
Chapter 8 of Data Engineering for Beginners opens with a statement that should be obvious but apparently is not: even the best pipelines and storage systems are meaningless if the data they deliver is garbage.
You ran the analysis. Got your numbers. Built a model. Now you need to show people what you found. And here is where most data people trip up. They pick the wrong chart, overload it with details, and the audience walks away confused.
You built your data pipelines. They work on your laptop. Now what? Chapter 10 of Data Engineering with Python by Paul Crickard covers the part everyone eventually has to face: getting your pipelines out of development and into production.
Chapter 2 showed Pythonistas how to pick up R. Chapter 3 flips the script. Now it’s the R user’s turn to step into Python territory. Rick Scavetta writes this one, and he does a good job easing R folks into a world that feels messier at first glance.
After NER and text classification, Funderburk moves to the third building block: sentiment analysis. Then she starts putting all the pieces together into a multi-agent architecture. This is where the chapter gets really interesting.
Data governance is one of those topics that sounds boring until you realize nobody can find anything in your data platform. Then it becomes very interesting very fast.
This is Part 2 of 2 for Chapter 10. In Part 1 we covered how to pick the right model and looked at regression. Now we get into the rest: time series, classification, clustering, and association analysis.
Chapter 8 is where Funderburk says: enough with single pipelines. Time to build tools. And then make an agent pick which tool to use.
You built a data pipeline. It is idempotent, uses atomic transactions, and has version control. It is production ready. But can you tell when it breaks?
This is Part 2 of Chapter 7, continuing from batch and streaming basics.
In Part 1, we covered how batch and streaming pipelines move data around. But here is the thing: having a pipeline is one thing. Making sure all its parts run in the right order, at the right time, without you babysitting it? That is orchestration. And this is where Chapter 7 gets really practical.
In Part 1 we covered R basics: setting up your environment, installing packages, working with tibbles, and understanding R’s type system. Now we get to the good stuff. Lists, factors, finding things in your data, and the iteration patterns that make R feel so different from Python.
In Part 1 we built a FastAPI app, Dockerized it, and locked it down with API keys. That is the “maximum control” path. It works great, but it requires a lot of boilerplate. Part 2 covers two things: automating the whole thing with CI/CD, and a completely different approach that makes most of that boilerplate disappear.
Chapter 9 is the one where Adi Wijaya zooms out from data pipelines and asks: okay, but who can access what, and how do we keep this whole thing organized? If the previous chapters taught you how to build things in GCP, this one teaches you how to not let those things turn into a security and management mess.
This is Part 1 of Chapter 7. Part 2 covers orchestration and transformations.
Chapter 7 of Data Engineering for Beginners is probably where things start feeling real. You stop talking about storage and tables and start talking about how data actually moves. And the answer is: through pipelines.
You have data. You have a question. But which model do you actually use?
Chapter 10 of “Data Science Foundations” by Stephen Mariadas and Ian Huke is the biggest chapter in the book. So big I split this retelling into two parts. This is Part 1. It covers the types of analytics, understanding your data and hypothesis, and how to pick the right model.
You’ve been building data pipelines for several chapters now. They work. They move data. But here’s the problem: none of them have version control. If you break something, there’s no going back. Chapter 8 of Data Engineering with Python by Paul Crickard fixes that. It introduces the NiFi Registry, a sub-project of Apache NiFi that handles version control for your data pipelines.
Chapter 2 is where the book gets hands-on. Rick Scavetta takes the wheel and walks Python developers through R. Not from scratch, but with the assumption you already know how to code. The chapter is big, so I split it into two posts. This is the first half.
Part 1 covered the ML basics: what supervised and unsupervised learning are, how a simple model gets trained, and why data engineers should care about ML at all. Now in Part 2, Adi Wijaya moves into the GCP tools that make ML actually work in production. This is where theory meets infrastructure.
Math scares people. I get it. You hear “standard deviation” and suddenly you are back in high school staring at the board. But here’s the thing. Chapter 9 of Data Science Foundations by Stephen Mariadas and Ian Huke covers the math basics you actually need for data science. And none of it is that hard.
Chapter 6 is where the book zooms out from “how to design one database” to “where does all this data actually live in a real company.” The answer: it depends on what you are trying to do with it.
Chapter 7 of Laura Funderburk’s book is where the rubber meets the road. You built a RAG pipeline in Chapter 6. Now you need to ship it. Get it out of a notebook and into something that real users can hit with HTTP requests.
You built a pipeline. It works on your machine. It runs on a schedule. Data goes in, data comes out. Ship it, right?
Chapter 1 is titled “In the Beginning” and it’s written by Rick Scavetta. He opens with a tongue-in-cheek Dickens reference, saying it’s just the best of times for data science. But to understand where we are, we need to look at where Python and R came from. Their origin stories explain why they feel so different today.
The previous chapters taught you the individual tools. Python, NiFi, Airflow, databases, data cleaning. Chapter 6 of Data Engineering with Python by Paul Crickard puts them all together into one real project.
Chapter 8 is the one where Adi Wijaya finally brings up the topic every data engineer either loves or dreads: machine learning. And honestly, he does a good job of calming down both camps. If you are excited about ML, great. If you think it has nothing to do with your job, think again. This chapter shows why ML and data engineering are way closer than most people realize.
You know that feeling when you buy fresh ingredients for dinner, and then spend 80% of your time washing, cutting, and peeling? The actual cooking takes 20 minutes. Data science is exactly like that. The cooking is the model. The prep work is this chapter.
In Part 1, we covered how Funderburk moves from Jupyter notebooks to a production-ready project structure. Docker, uv, SuperComponents, dual Elasticsearch. Now comes the part that actually tells you if your RAG pipeline is any good: systematic evaluation with RAGAS and continuous monitoring with Weights and Biases.
This is Part 2 of Chapter 5, continuing from data modeling basics.
If Part 1 was about drawing the blueprint, Part 2 is about keeping the building from falling apart. Normalization is one of those topics that sounds academic until you hit a real bug caused by duplicate data. Then it clicks fast.
The preface of “Python and R for the Modern Data Scientist” sets up the whole book in a few pages. And it does something rare for a tech book. It actually defines what it means by its own title.
I picked up “Python and R for the Modern Data Scientist” by Rick J. Scavetta and Boyan Angelov a while back. It’s an O’Reilly book from 2021, and it caught my eye because it doesn’t pick sides in the Python vs R debate. Instead, it argues you should use both.
You can build the best pipeline in the world. You can read files, write to databases, schedule everything with Airflow. But if the data going through that pipeline is messy, none of it matters.
This is post 10 in my Data Engineering with AWS retelling series.
Part 1 covered batch ingestion – pulling data from databases into S3 on a schedule. But not all data waits politely for a nightly load. IoT sensors, vehicle telemetry, live gameplay events, social media mentions – this data streams in continuously and often needs to be processed in near-real-time.
You spend weeks building pipelines, modeling data, setting up orchestration. Everything works. Data lands in BigQuery clean and on time. And then someone from the business side asks: “So… where do I see the numbers?” That is exactly where Chapter 7 picks up. All that upstream work has to end somewhere useful, and for most organizations that somewhere is a dashboard.
This is Part 1 of Chapter 5. Part 2 covers normalization and design best practices.
Chapter 5 of Data Engineering for Beginners by Chisom Nwokwu is about database design. And honestly, this is where things start to feel real. The previous chapters gave us SQL and database basics. Now we are drawing blueprints.
You have a great hypothesis. Your stakeholders are on board. But none of it matters without the right data.
Chapter 7 of “Data Science Foundations” by Stephen Mariadas and Ian Huke is about sourcing. Where do you get data? How do you collect it? How do you know if it is any good?
Chapter 6 is where the book shifts gears. Hard. Funderburk basically says: “Cool, you built a RAG pipeline. It works on your laptop. Now what?”
In Part 1 we covered Pub/Sub and how messages flow between publishers and subscribers. Now comes the fun part: what do you actually do with those messages once you have them? That’s where Apache Beam and Dataflow come in.
Chapter 6 of Data Science Foundations by Stephen Mariadas and Ian Huke is about something that sounds boring but really is not. Properties of data. What kind of data are you working with? And why does it matter so much?
In the last post we learned the rules for building custom Haystack components. Now Funderburk puts those rules to work on a real problem: building a pipeline that creates a knowledge graph from your documents and then generates synthetic test questions from that graph.
This is Part 2 of Chapter 4, continuing from the SQL basics.
In Part 1 we covered how to pull data from one table. Filter it, sort it, count it. But real databases have many tables. Customers in one, orders in another, products in a third. The interesting stuff happens when you combine them.
Most data pipelines start with a database. Most of them end with one too. Chapter 4 of Paul Crickard’s book is about connecting Python to databases and moving data between them. If the previous chapter was about flat files, this one is where things get real.
Chapter 5 is where Funderburk says: stop being a user of Haystack. Start being an architect. Up until now, the book has been about plugging together existing components. Now you learn to build your own.
Chapter 6 is where Adi Wijaya switches gears from batch to real-time. After spending Chapters 3 through 5 on batch pipelines with BigQuery, Cloud Composer, and Dataproc, now it is time to talk about streaming data. Two GCP services carry this chapter: Pub/Sub and Dataflow. This post covers the streaming concepts and Pub/Sub. Dataflow gets its own post in Part 2.
You got a data science project. Great. But before you touch any data, before you write a single line of code, you need to stop and think. That is what Chapter 5 of “Data Science Foundations” by Stephen Mariadas and Ian Huke is about. The discovery phase. The part most people want to skip. And it is the part that saves you from wasting months on something that never had a chance.
Chapter 3 is where Crickard moves from setup to actual work. You installed all those tools in Chapter 2. Now you use them. The chapter covers one of the most fundamental tasks in data engineering: getting data out of text files and into something useful.
This is Part 1 of Chapter 4. Part 2 covers joins and advanced queries.
Chapter 4 is where Nwokwu puts SQL in your hands. No more theory. You write queries, you get results, you learn by doing. If Chapter 3 was about understanding what databases are, this chapter is about talking to them.
Chapter 1 was all theory. Now it’s time to actually install stuff. Chapter 2 of Data Engineering with Python by Paul Crickard is a setup chapter. You install the tools, configure them, and make sure everything talks to each other.
In Part 1 we set up a Dataproc cluster, got familiar with HDFS, and touched on what a data lake actually is. Now it is time to get into the real work: writing PySpark code, understanding RDDs, moving data between HDFS, GCS, and BigQuery, and learning how to actually submit Spark jobs to Dataproc.
Imagine your company asks you to build a model that predicts health outcomes for people. Sounds great, right? Better treatments, healthier population, maybe even lower costs. But what if your health data gets shared? What if your insurance premiums go up because of something the model found? What if you get denied a service?
Chapter 3 is where things get real. You stop talking about data in the abstract and start working with the thing that actually holds it: databases. If you plan to do any data engineering at all, this is where your daily life begins.
In the last post we built a naive RAG pipeline. It works, but it has a blind spot: it only understands meaning, not exact words. Search for error code “ERR-4052” and the semantic retriever might miss the one document that contains that exact string. This is the vocabulary mismatch problem, and hybrid RAG is how you fix it.
Chapter 5 is where things get interesting if you come from a traditional database background. We are leaving the nice structured world of BigQuery and entering the territory of raw files, distributed storage, and the Hadoop ecosystem. Welcome to the data lake.
Chapter 2 was about stakeholders. Now Chapter 3 asks a very practical question: how do you actually get a data science project done?
Chapter 4 is where you stop reading about components and actually start wiring them together. Laura Funderburk calls it “Bringing Components Together,” and that’s exactly what it is. You take all those building blocks from Chapter 3 and connect them into working pipelines.
Chapter 1 was about understanding data itself. Chapter 2 answers the bigger question: what do data engineers actually do with it?
Chapter 1 of Data Engineering with Python by Paul Crickard starts with the basics. What is data engineering? What do data engineers actually do? And how is it different from data science?
In the first part we got Cloud Composer running, wrote our first DAGs, and learned operators. This second part covers the stuff that separates beginner Airflow code from production-ready pipelines: variables, idempotent tasks, backfilling, sensors, and dataset-driven scheduling.
So I picked up Data Engineering with Python by Paul Crickard (Packt, 2020, ISBN: 978-1-83921-418-9) and decided to write up my study notes as I go through it. I’ve been working in IT for over 20 years, and data engineering keeps coming up everywhere. This book seemed like a good one to work through and share what I learn.
You built a model. It works. The numbers look great. But nobody uses it.
Because you forgot the people around it. Chapter 2 of “Data Science Foundations” by Stephen Mariadas and Ian Huke is about exactly that. Stakeholders. The humans who care about your project, fund it, get affected by it, or can shut it down.
In the first part we covered Haystack 2.0’s core ideas: components, pipelines, SuperComponents, and how hybrid retrieval works. Now let’s look at what happens when you hand these pipelines to an AI agent, plus the full catalog of component types Haystack gives you out of the box.
The book opens with a simple claim: data is the new oil. You’ve probably heard that phrase a hundred times. But Nwokwu doesn’t just drop the cliche and move on. She actually walks you through why that comparison holds up, starting from thousands of years ago.
I picked up “Data Engineering for Beginners” by Chisom Nwokwu (Wiley, 2026, ISBN: 9781394325412) a few weeks ago. I was looking for something that explains data engineering from scratch, without assuming you already know half the field. This book does exactly that.
Up until now in the book, we built BigQuery tables by hand, wrote queries in the console, and loaded data manually. That works for learning, but nobody does that in production. In production, you need things to run on their own, on schedule, without you babysitting them at 5 AM.
You probably watched Moneyball. Brad Pitt, baseball, small team beats the rich guys using numbers. Good movie. But what the movie really shows is something bigger. It shows what happens when you take data seriously. And that is basically what data science is about.
Chapter 3 of Laura Funderburk’s book is where the rubber meets the road. We stop talking theory and start looking at an actual framework you can use to build real NLP pipelines. That framework is Haystack 2.0 by a company called deepset.
This is post 9 in my Data Engineering with AWS retelling series.
You have your whiteboard architecture from Chapter 5. You know who your data consumers are and what they need. Now it is time to actually move data. Chapter 6 covers data ingestion – getting data from wherever it lives into your AWS data lake. This first part focuses on batch ingestion from databases and files. Part 2 covers streaming.
In Part 1 we loaded CSV files into BigQuery and built a simple warehouse from a MySQL export. Now the book throws a second scenario at us: bike-sharing data. More tables, daily batch loading, and a real question that every data engineer has to face sooner or later. How do you actually model your data so that business people can use it without calling you every five minutes?
I have been working in tech for over 20 years. Seen a lot of trends come and go. But data science is not a trend. It is here to stay. And honestly, I wanted a book that explains the whole thing from the ground up without assuming I already know everything.
Parts 1 and 2 of this chapter covered transformer architecture, the SLM/RLM split, context engineering strategies, and the Haystack + LangGraph hybrid architecture. Now Funderburk closes the chapter with two topics that every developer building LLM applications needs to understand: vector stores and the economics of inference.
In Part 1, we covered how transformers work and how models split into small language models (SLMs) and reasoning language models (RLMs). Now Funderburk shifts to a big question: how do you actually interact with these models in a reliable way?
Chapter 3 is where things get real. Up to now the book was setting the stage, explaining what data engineering is, showing you around GCP. Now Adi Wijaya says: okay, let’s actually build something. And the something is a data warehouse in BigQuery.
Chapter 2 is where Adi Wijaya starts showing what Google Cloud Platform actually has for data engineers. After the theory in Chapter 1, this one is about opening GCP for the first time and figuring out which services matter and which ones you can safely ignore for now.
Chapter 2 of Laura Funderburk’s book opens with the big picture of large language models. Where they came from, how they work inside, and where they are heading. If Chapter 1 was about pipelines, this chapter is about the models that sit at the center of those pipelines.
Chapter 1 starts with a confession most of us in the data world can relate to. Adi Wijaya says he used to think data was clean. Neatly organized, ready to go. Then he actually worked with data in real organizations and realized most of the effort goes into collecting, cleaning, and transforming it. Not the fun machine learning part. The plumbing part.
In Part 1 we covered the agentic reliability crisis, what data pipelines are, and why classic NLP techniques are being reborn as tools for AI agents. Now let’s get into the specifics: how tokenization and embeddings actually work, what LLMs are, and the two very different roles they play in modern agentic systems.
I just finished reading “Data Engineering with Google Cloud Platform” by Adi Wijaya (2nd edition, Packt Publishing, 2024) and I want to share what I learned. Not as a dry summary, but more like telling a friend what the book is about over coffee.
Chapter 1 of Laura Funderburk’s book opens with something I wish more people in the AI space would say out loud: the era of pure experimentation with LLMs is over. We’re past the “look what ChatGPT can do” stage. The real question now is: can you trust this thing in production?
So I just finished reading Building Natural Language and LLM Pipelines by Laura Funderburk, and I wanted to share what I learned. This is one of those books that bridges the gap between “I can make a ChatGPT wrapper” and “I can build production AI systems that actually work.”
This is post 8 in my Data Engineering with AWS retelling series.
You have learned about data engineering principles, data architectures, the AWS toolkit, and data governance. Now comes the part where it all comes together. Chapter 5 is about designing an actual data pipeline. Not writing code yet. Just thinking. Planning. Drawing on a whiteboard.
In Part 1, we covered the theory: what data security and governance mean, how catalogs prevent your lake from becoming a swamp, and the core AWS services for encryption and identity. Now it is time to put it into practice.
You can have the fastest data pipeline on the planet. You can have the slickest dashboards, the fanciest machine learning models, the most optimized Parquet files. None of it matters if your data gets stolen, mishandled, or dumped into a lake that nobody can navigate.
In Part 1 we covered how data gets into AWS. Now comes the good part: what do you actually do with it once it is there? This post covers the services for transforming raw data, orchestrating multi-step pipelines, and letting people query and visualize the results.
Chapter 3 is massive. It is basically a catalog of every AWS service a data engineer will touch, from getting data in to getting answers out. So I am splitting it into two posts. This first part covers how data gets into AWS – all the ingestion services, the streaming tools, and the physical devices AWS will literally ship to your door.
Chapter 1 gave us the “who” and “why” of data engineering. Now it is time for the “where.” Where does all that data actually live? How do organizations store, organize, and serve billions of rows of information so that someone on the business side can pull up a dashboard and make a decision before lunch?
If someone told you twenty years ago that data would become more valuable than oil, you would have laughed. But here we are. The most valuable companies on the planet are not drilling for crude. They are collecting, processing, and squeezing insights out of massive piles of data. And behind every one of those companies, there is a team of data engineers making it all work.
Every company today is drowning in data. Clicks, transactions, sensor readings, log files, social media posts. It just keeps coming. But raw data sitting in a pile is useless. The real magic happens when someone builds the pipes that move it, clean it, reshape it, and deliver it to the people who need it.
Previous: Elastic MapReduce: Running Hadoop in the AWS Cloud
We’ve covered a lot of ground in this series. From the basic blocks of HDFS to the real-time speeds of Flink and the limitless scale of the AWS cloud. After spending a lot of time with Sridhar Alla’s Big Data Analytics with Hadoop 3, I have a few final thoughts to share.
Previous: Mastering AWS for Big Data: EC2, S3, and EMR
In the last post, we looked at the basic building blocks of AWS: EC2 and S3. But if you’re trying to run a massive Hadoop or Spark cluster, you don’t really want to be manually installing software on hundreds of individual EC2 instances. That’s where Amazon EMR (Elastic MapReduce) comes in.
Previous: Comparing the Giants: AWS, Azure, and Google Cloud
We’ve talked about the “what” and the “why” of the cloud. Now it’s time for the “how.” Chapter 12 of Sridhar Alla’s book is a deep look at Amazon Web Services (AWS), which is essentially the playground where most big data pros spend their time.
Previous: Cloud Computing for Big Data: An Introduction
In the last post, we looked at the basic models of the cloud (IaaS, PaaS, and SaaS). Today, we’re talking about the “where” and the “who.” When you decide to move your big data to the cloud, you have to choose a deployment model and a provider.
Previous: Visualizing Big Data: Turning Numbers into Insight
We’ve spent this entire series talking about how to set up and run your own Hadoop cluster. But let’s be real: managing hardware is a pain. You have to buy servers, set up networking, worry about power outages, and pray that your hard drives don’t fail.
Previous: Flink Connectors and Event Time: Mastering the Stream
You’ve done the hard work. You’ve set up a Hadoop cluster, written MapReduce jobs, and built real-time pipelines in Spark and Flink. You have “insights.” But here’s the problem: nobody wants to look at a raw HDFS file or a console log.
We’ve reached the end of our walkthrough of Data Analytics for Finance Using Python. We’ve covered everything from basic stats to deep learning, and if there’s one thing I hope you’ve learned, it’s this: these are tools, not magic.
Previous: Stream Processing with Apache Flink: True Real-Time Analytics
In the last post, we looked at Flink’s DataStream API. Today, we’re tackling the big questions: How does Flink handle the messy reality of the real world? How does it talk to other systems? And how does it deal with data that shows up late?
So here we are. Sixteen posts later and I’m done retelling “Learning Systems Thinking” by Diana Montalion. Time to wrap it up with what actually stuck with me.
We’ve saved the big gun for last. In Chapter 15 of Data Analytics for Finance Using Python, we dive into Deep Learning with Long Short-Term Memory (LSTM) models.
Previous: Flink DataSet API: Transformations, Joins, and Aggregations
We’ve talked about how Spark handles streaming using micro-batches. It’s a great approach, but some people argue it’s not “true” streaming. If you need the absolute lowest latency possible, you want Apache Flink.
Stocks aren’t just driven by math; they’re driven by people. And people are emotional. In Chapter 14 of Data Analytics for Finance Using Python, we look at Natural Language Processing (NLP)—a way to turn human chatter into useful data.
This is Part 2 of 2 for Chapter 12. If you missed Part 1, go read it first. We covered how success is a system, enabling constraints, root causes, equalizing impact, knowledge flow, and the paradigm shift. Now we finish the chapter and the book.
Previous: Batch Analytics with Apache Flink: The New Challenger
In the last post, we got Flink up and running. Now, let’s actually do something useful with it. Chapter 8 of Sridhar Alla’s book focuses on the DataSet API, which is what you’ll use for all your batch processing needs.
They say a picture is worth a thousand words, and in the stock market, it’s worth even more. In Chapter 13 of Data Analytics for Finance Using Python, we step away from the raw numbers and look at how to actually see the data.
We are near the end of the book. Chapter 12 is called “Redefining Success” and it asks two questions that sound simple but are not. How do you know you are learning systems thinking? And what does success even mean when you look at the whole system?
Previous: Structured Streaming: The Modern Way to Handle Data Streams
We’ve spent a lot of time on Spark, and for good reason - it’s amazing. But if you’re serious about big data, you need to know about Apache Flink. In Chapter 8, Sridhar Alla introduces us to the technology that many experts consider the “true” successor to MapReduce for real-time processing.
If you’re looking for the heavyweight champion of classification models, you’ve probably heard of Support Vector Machines (SVM). In Chapter 12 of Data Analytics for Finance Using Python, we see why this model is a favorite for high-precision tasks.
Chapter 11 is about leadership. But not the kind you see on LinkedIn where someone posts a sunset photo and writes “leaders eat last.” Diana is talking about something very different. Systems leadership is about improving how knowledge flows through your organization. Not about your title, not about your authority, not about how many people report to you.
Previous: Real-Time Analytics with Spark Streaming
In the last post, we looked at DStreams, the original way to do streaming in Spark. But things move fast in the tech world. Spark 2.0 introduced Structured Streaming, a new way to handle real-time data that makes things even simpler and more reliable.
If you’ve ever heard someone say a stock’s price is “three standard deviations away from the mean,” they’re talking about Z-Scores. In Chapter 11 of Data Analytics for Finance Using Python, we explore how to use this tool to find the “weird” data points that might actually be opportunities.
This is Part 2 of Chapter 10 from “Learning Systems Thinking” by Diana Montalion. Part 1 covered what modeling is and different modeling approaches. Now we get into the practical stuff. How do you actually use modeling to solve real problems? Diana brings back the MAGO case study to show us.
Previous: Spark SQL and Aggregations: Joining Your Data at Scale
Up until now, we’ve mostly talked about batch processing - looking at data that’s already sitting in HDFS. But what if you need to know what’s happening right now? What if you’re tracking a stock price, monitoring a server for hacks, or following a trending hashtag on Twitter? That’s where Spark Streaming comes in.
We just talked about the F-Test for comparing risk (variance), but what if you want to know if two stocks actually earn different amounts on average? In Chapter 10 of Data Analytics for Finance Using Python, we look at the T-Test.
Chapter 10 is a big one, so I’m splitting it into two parts. This is Part 1 of 2.
Diana opens with a Donella Meadows quote that sets the tone for everything that follows: get your model out where people can see it, invite others to challenge it. That’s the whole chapter in one sentence, really. But of course there’s much more to unpack.
Previous: Batch Analytics with Apache Spark: Faster Than MapReduce
In the last post, we looked at why Spark is so fast. Today, we’re getting into the nitty-gritty of how to actually use it. If you’re a SQL fan, you’re going to love this. Chapter 6 of Sridhar Alla’s book spends a lot of time on Spark SQL, and for good reason - it’s where most of the work happens.
We all see patterns. You get sick and you think back. Was I stressed? Did I sleep enough? Was someone around me sick? You compare symptoms to last time. You adjust your schedule.
If you’re choosing between two stocks, you don’t just want to know which one has a higher return. you want to know which one is more likely to give you a heart attack. In Chapter 9 of Data Analytics for Finance Using Python, we look at the F-Test as a way to compare risk.
Previous: Statistical Computing with R and Hadoop
If you’ve been following this series, you know we’ve spent a lot of time on MapReduce. It’s the foundation of Hadoop, but let’s be honest: it can be slow and painful to write. That’s why Chapter 6 of Sridhar Alla’s book is such a breath of fresh air. It introduces Apache Spark, the technology that has effectively dethroned MapReduce for most big data tasks.
In the world of finance, nothing happens in a vacuum. A stock’s closing price isn’t just a random number; it’s influenced by the opening price, the daily high, the daily low, and a dozen other factors.
When you hear “feedback loop” you probably think about monitoring dashboards. Or autoscaling. Or maybe that annoying annual performance review your manager gives you. Diana Montalion says all of that is too narrow. Chapter 8 is about feedback loops for thinking. Not for servers.
Previous: Scientific Computing with Python and Hadoop
If Python is the general-purpose king of data science, R is the specialized wizard of statistics. While Python is great for building pipelines and apps, R was built by statisticians, for statisticians. In Chapter 5, Sridhar Alla shows us how to bring that statistical power to the massive datasets sitting in Hadoop.
Chapter 7 is where Diana shifts from “you” to “we.” Previous chapters were about your own thinking, your own reactions, your own learning. Now it’s time to think together. Because in systems, your brain alone is not enough.
Before you start building complex machine learning models, you need to understand the data you’re actually working with. In Chapter 7 of Data Analytics for Finance Using Python, we go back to the basics: Descriptive Statistics.
Previous: Advanced MapReduce: Joins and Filtering Patterns
Java and MapReduce are great for the heavy lifting, but when it comes to actually exploring data and building models, Python is where it’s at. Chapter 4 of Sridhar Alla’s book shifts the focus to how we can use Python’s massive ecosystem to analyze big data.
The book is called “Learning Systems Thinking.” We talked about systems. We talked about thinking. Chapter 6 is about the third word. Learning.
We’ve talked about random forests, but sometimes it’s better to look at the individual trees. In Chapter 6 of Data Analytics for Finance Using Python, we dive into the Decision Tree Classifier.
Previous: Deep Look at MapReduce: How Hadoop Processes Data
In the last post, we looked at the basics of MapReduce. But in the real world, your data is rarely in one single file. You usually have a few different datasets that you need to combine. This is where things get a little more complex - and a lot more interesting.
The stock market is a rollercoaster. One minute you’re up, the next you’re down, and trying to predict that movement is like trying to catch lightning in a bottle. In Chapter 5, we look at a tool designed to handle that chaos: the Random Forest.
This chapter hit close to home. If you ever rage-typed a Slack message, hit send, and then immediately wished you could unsend it, this one is for you.
Previous: SQL on Hadoop: Getting Started with Apache Hive
We’ve talked about Hive, but today we’re going under the hood. MapReduce is the engine that actually does the heavy lifting in Hadoop. Sridhar Alla’s third chapter is a deep look at how this framework takes a massive pile of data and turns it into something useful.
If you’re looking for a machine learning model that’s fast, efficient, and actually outshines more complex models in some cases, you need to look at Gaussian Naive Bayes (GNB).
Chapter 4 starts Part II of the book. And Part II hits different. Part I was about systems out there, in the world, in software. Part II turns the mirror around. Now we’re looking at ourselves.
Previous: The World of Big Data Analytics: Processes and Tools
If you’ve ever tried to write a MapReduce job just to count the number of lines in a file, you know it’s a lot of work. You have to write a Mapper, a Reducer, a Driver… it’s a whole thing.
In Chapter 3 of Data Analytics for Finance Using Python, we move from predicting exact prices to making a much simpler, but arguably more important, decision: Should I buy or sell?
Chapter 3 opens with a quote from Donald Berwick: “Every system is perfectly designed to get the results it gets.” Read that again. If your system produces bad results, it was designed to produce bad results. Maybe not on purpose. But the design got you there.
Previous: Setting Up Your Hadoop 3 Cluster: A Step-by-Step Guide
Now that we’ve got a cluster running, let’s talk about why we bother with all this complexity in the first place. Chapter 2 of Sridhar Alla’s book takes a step back to look at the big picture of data analytics.
Predicting the stock market is basically the Holy Grail of finance. Everyone wants to know what’s going to happen tomorrow. In Chapter 2, we look at a classic tool for this: the ARIMA model.
Chapter 2 opens with a quote from Fred Brooks: “Conceptual integrity is the most important consideration in system design.” Written decades ago. Still true. Maybe more true now than ever.
Previous: Getting Started with Hadoop 3: What’s New and Why It Matters
In the last post, we talked about all the cool new features in Hadoop 3. Now, let’s actually build something. Sridhar Alla’s book gives a solid walkthrough on setting up a single-node cluster. If you’re on Linux, this is pretty straightforward.
In the first chapter of Data Analytics for Finance Using Python, we get into the nitty-gritty of portfolio management using something called K-Means clustering.
Diana starts Chapter 1 with a warning. Reading a book about systems thinking will not teach you systems thinking. Just like reading a book about tennis will not teach you tennis. You have to go outside and play. Fair enough. But you still need to know the rules before you step on the court.
Previous: Big Data for the Rest of Us
Hadoop has been around for a while, but version 3 is where things get really interesting. If you’ve worked with Hadoop 1 or 2, you know it was solid but had some pain points. Sridhar Alla’s book kicks off by looking straight at what’s changed.
So, I’ve been thinking about how the stock market works and why it feels like such a gatekept club. It turns out, if you know a bit of Python, you can actually peek behind the curtain.
I just finished reading “Learning Systems Thinking” by Diana Montalion (O’Reilly, 2024, ISBN: 978-1-098-15133-1) and I want to share what I got from it. Chapter by chapter. Like a retelling with my own thoughts mixed in.
So, you’ve heard about big data. It’s everywhere. But how do you actually handle it? If you’re looking for the OG of big data platforms, you’re looking at Hadoop. And honestly, it’s still the foundation for almost everything we do in data today.