Chapter 5: Real-Time Data With Amazon Kinesis
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Real time
Building an End-to-End Big Data Pipeline - Part 3
Batch processing is great for historical reports, but what if you need to know what’s happening right now? In the final part of Chapter 10, Neylson Crepalde shows us how to build a world-class Real-Time Pipeline on Kubernetes.
Real-Time Visualization With Elasticsearch and Kibana
Trino is great for querying your historical data on S3, but for real-time streams and text-heavy search, you need something different. In the second half of Chapter 9, Neylson Crepalde introduces the industry standard for real-time analytics: Elasticsearch and Kibana.
Real-Time Streaming With Apache Kafka - Part 2
Architecture is great, but let’s actually run some code. In the second half of Chapter 7, Neylson Crepalde walks us through setting up a multi-node Kafka cluster right on our local machine using Docker Compose.
Real-Time Streaming With Apache Kafka - Part 1
In the world of big data, “batch” is no longer enough. We need data the second it happens. Whether it’s tracking stock prices, monitoring website traffic, or detecting fraud, you need a system that can handle massive streams of events with zero downtime.