Home » Tags » Data-Pipelines-With-Apache-Airflow

Data pipelines with apache airflow

Airflow on GCP: GKE, BigQuery, and the Best Provider Package

Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901

GCP is the last cloud chapter and the most complete. Google has the largest Airflow provider package. The book walks through a full GKE deployment with Helm, not just diagrams. This is DIY Airflow, not Cloud Composer (covered in chapter 15).

Airflow on Azure: App Service, Synapse, and Blob Storage

Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901

Azure gets the same treatment AWS got in the previous chapter. Map Airflow components to managed services, survey the provider package, then build the movie-ranking pipeline again. Same goal, different cloud.

Airflow on AWS: Fargate, S3, Glue, and a Movie Ranking DAG

Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak and Julian deRuiter
ISBN: 9781617296901

Chapter 16 is where the cloud chapters get concrete. Harenslak and deRuiter map every Airflow component to an AWS service, then build a working movie-ranking pipeline with S3, Glue, and Athena.

Airflow Trigger Rules, XCom, and the Taskflow API Explained

Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901

The first half of chapter 5 was DAG shape and branching. Sections 5.4 through 5.6 answer: when exactly does a task run, how do tasks pass small bits of state, and can Python-heavy DAGs look like normal Python again?

Airflow Task Dependencies: Linear Chains, Fan-Out, Branching, and Conditionals

Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901

Chapter 5 is where DAGs stop looking like straight lines. The book covers dependency shapes, branching when your data source changes, and skipping tasks when a condition is not met. This post covers sections 5.1 through 5.3. Trigger rules, XCom, and Taskflow API are the next post.

Anatomy of an Airflow DAG: Your First Rocket Launch Pipeline

Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901

Chapter 1 was theory. Chapter 2 is hands-on. The book follows John, a rocket enthusiast who wants upcoming launch images on his machine. The Launch Library API returns JSON with image URLs. John’s plan: download metadata, fetch pictures, notify when done. That becomes your first real Airflow DAG.

Meet Apache Airflow: Why Data Pipelines Need a Workflow Orchestrator

Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian de Ruiter
ISBN: 9781617296901

I picked up Data Pipelines with Apache Airflow because I kept hearing the same complaint from data teams: the volume is fine, but the coordination is a mess. Megabytes turned into gigabytes per minute, and nobody had a clean way to run the same steps in order, on a schedule, with visibility when something broke. Chapter 1 is the orientation chapter. No install yet. Just: what is a pipeline, why draw it as a graph, and where Airflow sits in the zoo of workflow tools.

About

About BookGrill.net

BookGrill.net is a technology book review site for developers, engineers, and anyone who builds things with code. We cover books on software engineering, AI and machine learning, cybersecurity, systems design, and the culture of technology.

Know More

Category

Tags View all tags

Book-Retelling Data-Engineering Python Aws Apache-Airflow Chisom-Nwokwu Data-Science Data-Engineering-With-Python Paul-Crickard Bas-Harenslak Data-Pipelines-With-Apache-Airflow Julian-Deruiter Big-Data-on-Kubernetes Neylson-Crepalde Nlp Big-Data Algorithms Haystack Laura-Funderburk Llm Nlp-Llm-Pipelines Sam-Morley Adi-Wijaya Cloud-Computing Data-Engineering-Gcp Data-Structures Google-Cloud-Platform Bhagvan-Kommadi Go-Programming Golang Learn-Golang Data-Engineering-Aws Gareth-Eagar Data-Engineering-for-Beginners-Chisom-Nwokwu Data-Pipelines C-Plusplus-Programmers-Mindset Daniel-Vaughan Data-Science-Foundations Data-Science-the-Hard-Parts Ian-Huke Stephen-Mariadas Data-Engineering-for-Beginners Data-Analytics-for-Finance-Using-Python Devops Infrastructure-as-Code Nitin-Jaglal-Untwal Terraform Diana-Montalion Hashicorp Learning-Systems-Thinking Ravi-Mishra Systems-Thinking Terraform-Certification Apache-Spark Brian-Paul Data-Engineering-With-Aws Etl Docker Blockchain-and-Banking-How-Technological-Innovations-Are-Shaping-the-Banking-Industry-Pierluigi-Martino Book-Review Boyan-Angelov Pierluigi Martino Python-and-R-for-the-Modern-Data-Scientist R-Programming Rick-Scavetta Apache-Nifi Kubernetes Rag Apache-Kafka Streaming Elasticsearch Production Banking Data-Lake Machine-Learning S3 Classification Data-Governance Data-Visualization Hadoop Aws-Glue Bigquery Clustering Computational-Thinking Context-Engineering Cpp-Programmers-Mindset Data-Quality Data-Warehouse K-Means Kafka Langgraph Nifi Openmp Pandas Postgresql Pyspark Real-Time Sql Apache-Flink Azure