Finishing Data Engineering With AWS: Key Takeaways and Honest Review
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
Data pipelines
Final Thoughts on Data Engineering With Python
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I finished the full walkthrough. Fifteen chapters, one appendix, two big projects, and a lot of terminal time. Here’s where I landed.
Monitoring Data Pipelines in Apache NiFi
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
You built a pipeline that is idempotent, version controlled, and ready for production. Great. But production means things break when you are not looking. Elasticsearch goes down. A network hiccup kills a connection. Bad data slips through. Chapter 9 is about knowing when that happens without staring at a screen all day.
Batch vs Streaming Data Pipelines and Windowing
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 7 is where the book stops talking about storage and starts talking about movement. Data pipelines are the plumbing. They collect, ingest, process, store, and serve data. Sometimes on a schedule, sometimes the second an event happens.
Data Engineering With AWS: Why I'm Reading This Book
Book: Data Engineering with AWS: A Comprehensive Guide to Building Robust Data Pipelines
Author: Brian Paul
ISBN: Not listed in the source edition
What Is Data Engineering? ETL, Pipelines, and the Tools Behind the Scenes
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Chapter 1 of Paul Crickard’s book is the orientation lecture. No installs yet. No code. Just a clear picture of what data engineering is, who does it, and what tools show up again and again.
The Data Engineering Life Cycle Explained With an Oil Refinery Analogy
Book: Data Engineering for Beginners
Author: Chisom Nwokwu
ISBN: 9781394325412
Chapter 2 is where Data Engineering for Beginners stops talking about data in general and starts defining the job. Nwokwu answers the question everyone asks: what does a data engineer actually do all day?
Data Engineering With Python: A Hands-on Book Walkthrough
Book: Data Engineering with Python
Author: Paul Crickard
ISBN: 978-1-83921-418-9
Publisher: Packt (October 2020)
I’m starting a walkthrough of Data Engineering with Python by Paul Crickard. This is not a copy-paste summary. It’s my retelling of each chapter with notes on what stuck, what felt dated, and what still holds up.
Data Pipelines With Apache Airflow: A Chapter-by-Chapter Walkthrough
Book: Data Pipelines with Apache Airflow
Authors: Bas Harenslak, Julian deRuiter
ISBN: 9781617296901
I’m working through Data Pipelines with Apache Airflow and writing up each chapter as a blog post. Not a dry summary. More like a retelling with my own reactions, the parts that clicked, and the parts that made me pause.
Data Engineering With Python: Final Thoughts and Takeaways
That’s it. Fifteen chapters, seventeen posts, and one complete walkthrough of Paul Crickard’s Data Engineering with Python (Packt, 2020, ISBN: 978-1-83921-418-9).
Data Engineering With GCP Chapter 12 Part 2: Building CI/CD Pipelines on Google Cloud
In Part 1 we covered the theory behind CI/CD and ran through a basic Cloud Build exercise with a Python project. Unit tests ran automatically on every push. Broken code got caught before it reached production. Good stuff, but that was a simple calculator script. Now we need to connect this to real data engineering work.
Building a Production Data Pipeline - Study Notes From Data Engineering With Python Ch 11
You learned the individual tools. You learned the deployment strategies. Now Chapter 11 of Data Engineering with Python by Paul Crickard puts it all together. This is the chapter where you build a complete, production-grade data pipeline from start to finish.
Monitoring Data Pipelines - Study Notes From Data Engineering With Python Ch 9
You built a data pipeline. It is idempotent, uses atomic transactions, and has version control. It is production ready. But can you tell when it breaks?
Data Pipelines: Batch vs Streaming and When to Use Each
This is Part 1 of Chapter 7. Part 2 covers orchestration and transformations.
Chapter 7 of Data Engineering for Beginners is probably where things start feeling real. You stop talking about storage and tables and start talking about how data actually moves. And the answer is: through pipelines.
Production Pipeline Features - Study Notes From Data Engineering With Python Ch 7
You built a pipeline. It works on your machine. It runs on a schedule. Data goes in, data comes out. Ship it, right?
Building a 311 Data Pipeline - Study Notes From Data Engineering With Python Ch 6
The previous chapters taught you the individual tools. Python, NiFi, Airflow, databases, data cleaning. Chapter 6 of Data Engineering with Python by Paul Crickard puts them all together into one real project.
What Is Data Engineering? Study Notes From Data Engineering With Python Ch 1
Chapter 1 of Data Engineering with Python by Paul Crickard starts with the basics. What is data engineering? What do data engineers actually do? And how is it different from data science?
Data Engineering With Python: My Study Notes From Paul Crickard's Book
So I picked up Data Engineering with Python by Paul Crickard (Packt, 2020, ISBN: 978-1-83921-418-9) and decided to write up my study notes as I go through it. I’ve been working in IT for over 20 years, and data engineering keeps coming up everywhere. This book seemed like a good one to work through and share what I learn.
Data Engineering With GCP Chapter 4 Part 1: Automating Data Workflows With Cloud Composer
Up until now in the book, we built BigQuery tables by hand, wrote queries in the console, and loaded data manually. That works for learning, but nobody does that in production. In production, you need things to run on their own, on schedule, without you babysitting them at 5 AM.
Data Engineering With AWS Chapter 5: Architecting Data Engineering Pipelines
This is post 8 in my Data Engineering with AWS retelling series.
You have learned about data engineering principles, data architectures, the AWS toolkit, and data governance. Now comes the part where it all comes together. Chapter 5 is about designing an actual data pipeline. Not writing code yet. Just thinking. Planning. Drawing on a whiteboard.