Building an End-to-End Big Data Pipeline - Part 1

We have spent the last few weeks looking at individual tools like Spark, Airflow, and Kafka. But in the real world, these tools don’t live in isolation. They need to talk to each other to form a complete data pipeline.

In Chapter 10 of Big Data on Kubernetes, Neylson Crepalde pulls everything together. This is where we move from “playing with tools” to “building a platform.”

The Integration Challenge

To build a full pipeline, your tools need permissions to interact. On Kubernetes, this means setting up RBAC (Role-Based Access Control). We have to create a ServiceAccount for Airflow and give it a ClusterRole that allows it to manage SparkApplication resources.

If you don’t get the permissions right, your Airflow worker will try to start a Spark job and Kubernetes will simply say “Access Denied.”

Checking the Foundation

Before we write any pipeline code, we have to make sure our cluster is healthy. The book suggests a “pre-flight check”:

• Is the Spark Operator running?
• Is the Strimzi Operator (for Kafka) healthy?
• Are Trino and Elasticsearch reachable?

On a managed service like EKS, you can check all of this with a few kubectl get pods commands across different namespaces.

Infrastructure as Code

The beauty of this approach is that the entire platform is defined as code. Your Airflow deployment, your Spark configurations, and your Kafka topics are all just YAML files.

If you need to move your entire pipeline from AWS to Google Cloud, you just take those same YAML files and apply them to a GKE cluster. The infrastructure changes, but the logic stays the same.

In the next post, we’re going to walk through the actual implementation of a Batch Pipeline using the IMDB dataset. We’ll automate everything from the first download to the final SQL query.

Next: Building an End-to-End Big Data Pipeline - Part 2 Previous: Real-Time Visualization with Elasticsearch and Kibana

Book Details:

• Title: Big Data on Kubernetes: A practical guide to building efficient and scalable data solutions
• Author: Neylson Crepalde
• ISBN: 978-1-83546-214-0