Chapter 7: Serverless Data Processing with AWS Lambda

Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition

Previous: Orchestrating with Step Functions

Chapter 7 covers serverless data processing. Lambda is the main tool, but Paul starts with the broader serverless concept.

Serverless in plain terms

Serverless means you write code, upload it, and the cloud runs it when triggered. No servers to provision, patch, or scale.

Key ideas Paul covers:

• Event-driven - functions run in response to events (S3 upload, DynamoDB change, API call)
• Stateless - each invocation is independent; state lives in external storage
• Pay-per-use - billed by execution time and memory, not idle server hours
• Auto-scaling - concurrency scales with incoming events

He compares Lambda with Azure Functions and Google Cloud Functions. Fair, though this is an AWS book so the depth is on Lambda.

AWS Lambda basics

Lambda runs your code in response to triggers:

• S3 object created → run transform function
• Kinesis record → process and write to DynamoDB
• API Gateway request → validate and route data
• CloudWatch schedule → kick off a nightly cleanup

Supported languages: Python, Node.js, Java, Go, C#, Ruby. Python is the most common for data work.

Functions are stateless. Store persistent data in S3, DynamoDB, or RDS. Keep functions small and focused.

Building data processing functions

Paul gives a practical checklist:

1. Identify sources and events - what triggers your function?
2. Define logic - one function, one job (validate, transform, route)
3. Pick runtime - Python is the safe default for data teams
4. Use AWS SDKs - boto3 for S3, DynamoDB, Athena interactions
5. Handle errors - Lambda retries automatically, but configure dead-letter queues
6. Environment variables - keep secrets out of code
7. Tune execution time - right-size memory to reduce cost and cold starts
8. Integrate - connect with Step Functions, Glue, Kinesis
9. Test locally - SAM or Serverless Framework before deploying
10. Deploy and let it scale - Lambda handles concurrency

This is solid advice. The “one function, one job” principle is especially important. Don’t cram an entire ETL pipeline into one Lambda.

Orchestrating Lambda in pipelines

Section 7.4 connects Lambda to Step Functions and Data Pipeline. The pattern:

• Step Functions calls Lambda functions as workflow steps
• Lambda triggers Glue jobs or writes to Kinesis
• CloudWatch monitors function duration and errors

Paul’s example: Lambda extracts data, another Lambda transforms it, a third loads it to Redshift. Step Functions coordinates the sequence.

What I think

Decent chapter. The serverless overview in 7.1 is useful context even if you’ve used Lambda before. The function-building checklist in 7.3 is the most actionable section.

Gaps: no cold start discussion (matters for latency-sensitive pipelines), no Lambda limits (15-minute timeout, 10GB memory cap), and no cost comparison with Glue for batch transforms. A 5-minute Glue job might be cheaper than chaining 50 Lambda invocations.

Key takeaway

Lambda is for small, event-driven data tasks. Validate a file upload. Transform a single record. Trigger the next step in a pipeline. For heavy ETL, use Glue. For coordination, use Step Functions.

Previous: Orchestrating with Step Functions | Next: Monitoring and Logging