Home » Tags » Batch-Processing

Batch processing

Data Pipelines: Batch vs Streaming and When to Use Each

This is Part 1 of Chapter 7. Part 2 covers orchestration and transformations.

Chapter 7 of Data Engineering for Beginners is probably where things start feeling real. You stop talking about storage and tables and start talking about how data actually moves. And the answer is: through pipelines.

Data Engineering With AWS Chapter 6 Part 1: Ingesting Batch Data

This is post 9 in my Data Engineering with AWS retelling series.

You have your whiteboard architecture from Chapter 5. You know who your data consumers are and what they need. Now it is time to actually move data. Chapter 6 covers data ingestion – getting data from wherever it lives into your AWS data lake. This first part focuses on batch ingestion from databases and files. Part 2 covers streaming.

About

About BookGrill.net

BookGrill.net is a technology book review site for developers, engineers, and anyone who builds things with code. We cover books on software engineering, AI and machine learning, cybersecurity, systems design, and the culture of technology.

Know More

Category

Tags View all tags

Book-Retelling Data-Engineering Data-Science Python Nlp Haystack Laura-Funderburk Llm Nlp-Llm-Pipelines Adi-Wijaya Data-Engineering-Gcp Google-Cloud-Platform Data-Science-Foundations Ian-Huke Stephen-Mariadas Chisom-Nwokwu Data-Engineering-for-Beginners Data-Engineering-With-Python Paul-Crickard Aws Data-Engineering-Aws Gareth-Eagar Boyan-Angelov Python-and-R-for-the-Modern-Data-Scientist R-Programming Rick-Scavetta Data-Pipelines Rag Apache-Airflow Apache-Nifi Docker Apache-Spark Context-Engineering Data-Governance Langgraph Machine-Learning Apache-Kafka Bigquery Book-Review Data-Lake Data-Quality Etl Production Agentic-Ai Big-Data Cloud-Composer Cloud-Computing Data-Visualization Data-Warehouse Dataproc Deployment Elasticsearch Multi-Agent Ner Project Sentiment-Analysis Streaming-Data Transformers A2a-Protocol Analytics Aws-Glue Batch-Processing Beginners-Guide Career-Advice Career-Growth Ci-Cd Cicd Cloud-Build Custom-Components Data-Catalog Data-Cleaning Data-Engineer-Role Data-Modeling Data-Security Data-Transformation Data-Types Databases Dataflow Deepset Document-Stores Fastapi Future-Ai Gcp-Certification Hadoop Hybrid-Rag Iam Kinesis Knowledge-Graphs Mcp Metadata Microservices Model-Selection Nvidia-Nims Pandas Partitioning Pipelines Postgresql Project-Management Pub-Sub Pyspark