Finishing Data Pipelines With Apache Airflow: Key Takeaways
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Twenty-five posts later, I have read the whole book. Here is where I landed.
Apache airflow
Running the Book's Code, Airflow 2 Imports, and Prometheus Metrics
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
The main chapters end at GCP. Three appendices handle the practical stuff: how to run the code, how imports changed in Airflow 2, and how to map Airflow metrics for Prometheus.
Airflow on GCP: GKE, BigQuery, and the Best Provider Package
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
GCP is the last cloud chapter and the most complete. Google has the largest Airflow provider package. The book walks through a full GKE deployment with Helm, not just diagrams. This is DIY Airflow, not Cloud Composer (covered in chapter 15).
Airflow on Azure: App Service, Synapse, and Blob Storage
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
Azure gets the same treatment AWS got in the previous chapter. Map Airflow components to managed services, survey the provider package, then build the movie-ranking pipeline again. Same goal, different cloud.
Airflow on AWS: Fargate, S3, Glue, and a Movie Ranking DAG
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
Chapter 16 is where the cloud chapters get concrete. Harenslak and deRuiter map every Airflow component to an AWS service, then build a working movie-ranking pipeline with S3, Glue, and Athena.
Airflow in the Clouds: Deployment Strategy and Managed Services
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901
Chapter 15 is the bridge chapter. You spent the last few chapters building DAGs, custom operators, and a real project. Now the book asks a harder question: where does all of this actually run?
NYC Fastest Route: A Full Airflow Project Walkthrough
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Securing Apache Airflow | Next: Airflow in the Clouds
Securing Apache Airflow: RBAC, Encryption, and Secrets
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Operating Airflow in Production (Part 2) | Next: NYC Fastest Route Airflow Project
Operating Airflow in Production: Logging, Monitoring, and Alerts
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Operating Airflow in Production (Part 1) | Next: Securing Apache Airflow
Operating Airflow in Production: Architectures and Executors
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Airflow Best Practices (Part 2) | Next: Operating Airflow in Production (Part 2)
Airflow Best Practices: Data Efficiency and Resource Management
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Airflow Best Practices (Part 1) | Next: Operating Airflow in Production (Part 1)
Airflow Best Practices: Clean DAGs and Reproducible Tasks
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
Previous: Running Airflow Tasks in Containers (Part 2) | Next: Airflow Best Practices (Part 2)
KubernetesPodOperator: Running Airflow Tasks on Kubernetes
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Running Airflow Tasks in Containers (Part 1) | Next: Airflow Best Practices (Part 1)
Why Run Airflow Tasks in Docker Containers
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Testing Airflow DAGs (Part 2) | Next: Running Airflow Tasks in Containers (Part 2)
Testing Templated Operators, Postgres Containers, and DAG Integration
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Testing Airflow DAGs (Part 1) | Next: Running Airflow Tasks in Containers (Part 1)
Testing Airflow DAGs: Integrity Checks, CI/CD, and Mocking
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Building Custom Airflow Components (Part 2) | Next: Testing Airflow DAGs (Part 2)
Building a Data Engineering Lab: NiFi, Airflow, PostgreSQL, and Elasticsearch
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 told you what data engineering is. Chapter 2 asks you to build the whole workshop on one Linux machine.
Custom Airflow Operators, Sensors, and Python Packaging
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Building Custom Airflow Components (Part 1) | Next: Testing Airflow DAGs (Part 1)
What Is Data Engineering? ETL, Pipelines, and the Tools Behind the Scenes
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 of Paul Crickard’s book is the orientation lecture. No installs yet. No code. Just a clear picture of what data engineering is, who does it, and what tools show up again and again.
Building Custom Airflow Hooks: From PythonOperator to MovielensHook
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Airflow and External Systems | Next: Building Custom Airflow Components (Part 2)
Connecting Airflow to AWS, Postgres, and Docker for Real Pipelines
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Triggering Airflow Workflows | Next: Building Custom Airflow Components (Part 1)
Triggering Airflow Workflows: Sensors, DAG Chains, and REST API
Data Pipelines with Apache Airflow by Bas Harenslak and Julian deRuiter (ISBN: 9781617296901)
Previous: Airflow Branching and the TaskFlow API | Next: Airflow and External Systems
Airflow Trigger Rules, XCom, and the Taskflow API Explained
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
The first half of chapter 5 was DAG shape and branching. Sections 5.4 through 5.6 answer: when exactly does a task run, how do tasks pass small bits of state, and can Python-heavy DAGs look like normal Python again?
Airflow Task Dependencies: Linear Chains, Fan-Out, Branching, and Conditionals
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 5 is where DAGs stop looking like straight lines. The book covers dependency shapes, branching when your data source changes, and skipping tasks when a condition is not met. This post covers sections 5.1 through 5.3. Trigger rules, XCom, and Taskflow API are the next post.
Templating Airflow Tasks: Jinja, Context Variables, and Hooks
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 3 showed {{ds}} in a curl command. Chapter 4 explains why that works, what else you can template, and how operators talk to Postgres without you writing connection boilerplate.
Scheduling in Airflow: Cron, Intervals, Backfill, and Idempotent Tasks
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 2 ended with @daily and a glimpse of the tree view. Chapter 3 goes deep on scheduling: how Airflow slices time, how to process only today’s data instead of everything every day, and what makes a task safe to rerun.
Anatomy of an Airflow DAG: Your First Rocket Launch Pipeline
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
Chapter 1 was theory. Chapter 2 is hands-on. The book follows John, a rocket enthusiast who wants upcoming launch images on his machine. The Launch Library API returns JSON with image URLs. John’s plan: download metadata, fetch pictures, notify when done. That becomes your first real Airflow DAG.
Meet Apache Airflow: Why Data Pipelines Need a Workflow Orchestrator
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901
I picked up Data Pipelines with Apache Airflow because I kept hearing the same complaint from data teams: the volume is fine, but the coordination is a mess. Megabytes turned into gigabytes per minute, and nobody had a clean way to run the same steps in order, on a schedule, with visibility when something broke. Chapter 1 is the orientation chapter. No install yet. Just: what is a pipeline, why draw it as a graph, and where Airflow sits in the zoo of workflow tools.
Data Pipelines With Apache Airflow: A Chapter-by-Chapter Walkthrough
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
I’m working through Data Pipelines with Apache Airflow and writing up each chapter as a blog post. Not a dry summary. More like a retelling with my own reactions, the parts that clicked, and the parts that made me pause.
Building an End-to-End Big Data Pipeline - Part 2
In our last post, we checked the infrastructure. Now, let’s build the actual pipeline. Neylson Crepalde uses the IMDB dataset to demonstrate a professional batch workflow.
Deploying the Big Data Stack on Kubernetes - Part 2
In my last post, we got Spark running natively on Kubernetes. Now, it’s time to bring in the conductor (Airflow) and the nervous system (Kafka). This is where your cluster starts to feel like a real data platform.
Orchestrating Pipelines With Apache Airflow - Part 2
In the last post, we got Airflow running. Now, let’s talk about how to actually use it. The heart of Airflow is the DAG—the Directed Acyclic Graph.
Orchestrating Pipelines With Apache Airflow - Part 1
If Spark is the engine, then Apache Airflow is the conductor. In a modern data stack, you rarely have just one job running in isolation. You have ingestion, cleaning, processing, and delivery—and they all have to happen in a specific order.
The Tools of the Modern Data Stack
We’ve talked about the architecture, but what about the actual tools? To build a modern data lakehouse on Kubernetes, you need a specific set of tools that can handle scale, automation, and speed.
Data Engineering With Python: Final Thoughts and Takeaways
That’s it. Fifteen chapters, seventeen posts, and one complete walkthrough of Paul Crickard’s Data Engineering with Python (Packt, 2020, ISBN: 978-1-83921-418-9).
Deploying Data Pipelines - Study Notes From Data Engineering With Python Ch 10
You built your data pipelines. They work on your laptop. Now what? Chapter 10 of Data Engineering with Python by Paul Crickard covers the part everyone eventually has to face: getting your pipelines out of development and into production.
Pipeline Orchestration With Airflow, DAGs, and Data Transformations
This is Part 2 of Chapter 7, continuing from batch and streaming basics.
In Part 1, we covered how batch and streaming pipelines move data around. But here is the thing: having a pipeline is one thing. Making sure all its parts run in the right order, at the right time, without you babysitting it? That is orchestration. And this is where Chapter 7 gets really practical.
Building a 311 Data Pipeline - Study Notes From Data Engineering With Python Ch 6
The previous chapters taught you the individual tools. Python, NiFi, Airflow, databases, data cleaning. Chapter 6 of Data Engineering with Python by Paul Crickard puts them all together into one real project.
Building Your Data Engineering Setup - Study Notes From Data Engineering With Python Ch 2
Chapter 1 was all theory. Now it’s time to actually install stuff. Chapter 2 of Data Engineering with Python by Paul Crickard is a setup chapter. You install the tools, configure them, and make sure everything talks to each other.
Data Engineering With GCP Chapter 4 Part 2: Airflow Scheduling, Idempotency, and Sensors
In the first part we got Cloud Composer running, wrote our first DAGs, and learned operators. This second part covers the stuff that separates beginner Airflow code from production-ready pipelines: variables, idempotent tasks, backfilling, sensors, and dataset-driven scheduling.
Data Engineering With GCP Chapter 4 Part 1: Automating Data Workflows With Cloud Composer
Up until now in the book, we built BigQuery tables by hand, wrote queries in the console, and loaded data manually. That works for learning, but nobody does that in production. In production, you need things to run on their own, on schedule, without you babysitting them at 5 AM.