Elasticsearch

• Jun 19, 2026
• data-engineering

Building a 311 Data Pipeline With NiFi and Kibana

Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9

Chapter 6 is where the book stops feeling like a tutorial and starts feeling like a real project. Crickard builds a pipeline that pulls citizen service requests from SeeClickFix (think 311-style complaints: graffiti, potholes, abandoned cars, needles, people ignoring social distancing during COVID). The data lands in Elasticsearch. Then you build a Kibana dashboard on top. He says he still runs this pipeline every 8 hours. That detail matters. This is not a throwaway demo.

• Jun 17, 2026
• data-engineering

Working With Databases: PostgreSQL, Elasticsearch, Airflow, and NiFi Pipelines

Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9

Chapter 4 is where the book stops playing with files on disk and starts working with databases. For most data engineers, this is the daily bread.

• Mar 26, 2026
• Big Data

Building an End-to-End Big Data Pipeline - Part 3

Batch processing is great for historical reports, but what if you need to know what’s happening right now? In the final part of Chapter 10, Neylson Crepalde shows us how to build a world-class Real-Time Pipeline on Kubernetes.

• Mar 23, 2026
• Big Data

Real-Time Visualization With Elasticsearch and Kibana

Trino is great for querying your historical data on S3, but for real-time streams and text-heavy search, you need something different. In the second half of Chapter 9, Neylson Crepalde introduces the industry standard for real-time analytics: Elasticsearch and Kibana.

• Feb 21, 2026
• software-engineering

Monitoring Data Pipelines - Study Notes From Data Engineering With Python Ch 9

You built a data pipeline. It is idempotent, uses atomic transactions, and has version control. It is production ready. But can you tell when it breaks?

• Feb 16, 2026
• software-engineering

Working With Databases - Study Notes From Data Engineering With Python Ch 4

Most data pipelines start with a database. Most of them end with one too. Chapter 4 of Paul Crickard’s book is about connecting Python to databases and moving data between them. If the previous chapter was about flat files, this one is where things get real.

• Feb 14, 2026
• software-engineering

Building Your Data Engineering Setup - Study Notes From Data Engineering With Python Ch 2

Chapter 1 was all theory. Now it’s time to actually install stuff. Chapter 2 of Data Engineering with Python by Paul Crickard is a setup chapter. You install the tools, configure them, and make sure everything talks to each other.

• Jan 17, 2019
• Big Data

Flink Connectors and Event Time: Mastering the Stream

Previous: Stream Processing with Apache Flink: True Real-Time Analytics

In the last post, we looked at Flink’s DataStream API. Today, we’re tackling the big questions: How does Flink handle the messy reality of the real world? How does it talk to other systems? And how does it deal with data that shows up late?