Local Kubernetes with Kind

Reading about architecture is one thing, but actually seeing a cluster run is where it sticks. In the third chapter of Big Data on Kubernetes, Neylson Crepalde moves from theory to practice.

If you are just starting out, you don’t need a massive AWS bill to learn Kubernetes. You can run it right on your laptop using a tool called Kind.

Your Universal Remote: kubectl

Before you do anything else, you need kubectl. This is the command-line tool you’ll use to talk to any Kubernetes cluster, whether it’s on your machine or in the cloud.

Installing it is straightforward:

• macOS: brew install kubectl
• Linux: Use curl to download the binary from the official Kubernetes site.
• Windows: choco install kubernetes-cli

Once it’s installed, run kubectl version --client to make sure it’s ready to go.

What is Kind?

Kind stands for “Kubernetes in Docker.” It essentially runs a Kubernetes cluster by spinning up Docker containers that act as nodes. It’s light, fast, and because it follows the official Kubernetes standards, what works here will almost certainly work in production.

Spinning up your first cluster

Assuming you already have Docker installed, setting up a cluster with Kind is almost too easy.

1. Install Kind: Similar to kubectl, you can use Homebrew, curl, or Chocolatey.
2. Create the cluster: Open your terminal and run:

   kind create cluster
   

That’s it. Kind will download the necessary images and configure a single-node cluster for you. It usually takes less than two minutes.

Checking your work

To verify that everything is running correctly, use this command:

kubectl cluster-info

If you see a URL for the Kubernetes control plane, you are officially a cluster administrator!

Why Kind is great for data engineers

When I’m testing a new Spark job or an Airflow DAG, I don’t want to wait for a cloud provider to provision nodes. Kind allows me to iterate locally, find the bugs, and fix the configuration before I ever touch the production environment. It’s the ultimate “sandbox” for data platform engineering.

But eventually, you will need more power. In the next post, we’ll look at how to scale up to a managed cluster using Amazon EKS.

Next: Scaling to the Cloud with Amazon EKS Previous: Decoding Kubernetes Architecture - Part 2

Book Details:

• Title: Big Data on Kubernetes: A practical guide to building efficient and scalable data solutions
• Author: Neylson Crepalde
• ISBN: 978-1-83546-214-0