Chapter 5: Real-Time Data with Amazon Kinesis

Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition

Previous: Building Pipelines with AWS Glue

Chapter 5 switches from batch to streaming. Paul covers the three Kinesis services and when each one fits.

Batch vs real-time

Paul starts with the core difference. Batch processing collects data over time and processes it in chunks. Real-time processing handles data as it arrives, usually within milliseconds or seconds.

Use cases he mentions: fraud detection in finance, patient monitoring in healthcare, IoT sensor data, log processing. All situations where waiting for a nightly batch job is too slow.

Kinesis Data Streams

Streams is the ingestion layer. Producers (apps, sensors, log agents) push records into a stream. Data gets partitioned into shards, each with its own throughput capacity.

Consumers read from shards in real time. You build them with the AWS SDK or Kinesis Client Library (KCL). Common patterns: anomaly detection, live dashboards, alert triggers.

Streams integrates with Lambda, CloudWatch, and Firehose. You get full control over how data is processed.

When to use it: you need custom processing logic on streaming data.

Kinesis Data Firehose

Firehose is the “just deliver it” option. You point it at a destination (S3, Redshift, OpenSearch, Splunk) and it handles delivery automatically. No consumer code required.

Key features:

• Auto-scaling based on incoming volume
• Optional Lambda transform before delivery
• Near-real-time delivery (not instant, but close)
• Built-in retry and buffering

When to use it: you want streaming data in S3 or Redshift without writing consumer apps.

Kinesis Data Analytics

Analytics lets you run SQL queries on streaming data. Write standard SQL, point it at a Kinesis source, and get results in real time.

It supports windowed computations (tumbling, sliding windows) which matter for time-series analysis. Output can go to Lambda, S3, or another Kinesis stream.

Paul notes the interactive dev environment. You can tweak SQL and see results live. That’s genuinely useful for iterating on streaming queries.

When to use it: your team knows SQL but doesn’t want to write streaming apps in Java or Python.

How they connect

A typical setup:

Sensors → Kinesis Streams → Lambda (process) → Firehose → S3
App logs → Kinesis Streams → Analytics (SQL) → Lambda (alert)

Or skip Streams entirely if you just need delivery:

App logs → Firehose → S3 → Glue (batch ETL) → Redshift

What I think

Clear chapter, good structure. Each service gets a defined role. The batch vs real-time framing in 5.1 is the most useful part for beginners.

What’s thin: no mention of Kinesis pricing (shard hours add up), no comparison with MSK (Managed Kafka), and no discussion of exactly-once vs at-least-once delivery semantics. Those matter when you go to production.

Also, Kinesis Analytics has been rebranded and evolved. Check current AWS docs for the latest service name and capabilities.

Key takeaway

Streams for custom processing. Firehose for managed delivery. Analytics for SQL on live data. Pick based on how much control you need.

Previous: Building Pipelines with AWS Glue | Next: Orchestrating with Step Functions