Q1
GCP Data Engineering · Chapter 01
Fundamentals of Data Engineering
Learn by poking at it. Explore the data life cycle, flip between a warehouse and a lake, sort real scenarios into ETL or ELT, and split a file across machines. Then test yourself.
0 of – checks answered
01
The data life cycle
Data behaves like water in pipes. It starts upstream where it is created and flows downstream to where it is used. Each arrow is its own extract-transform-load step. Pick a stage.
Where data is born
The front-end app is the first upstream. Its database is tuned to record each transaction as fast as possible, not to answer big questions across many systems.
- Typical format
- Rows in a transactional (OLTP) database
- On GCP
- Cloud SQL · Spanner · Firestore
- Question it serves
- "Did this payment go through?"
- Data engineer focus
- Good to know
Everything, raw, in one place
Copies from many databases land here as plain files. No schema is needed yet, so anything can be stored first and understood later.
- Typical format
- Files: CSV, JSON, Avro, Parquet, images
- On GCP
- Cloud Storage · Dataproc
- Question it serves
- "Keep it all. We'll decide what matters later."
- Data engineer focus
- Core
Modelled for the business
Once data is known to be valuable, it gets a schema and a model built around the questions the business wants answered, then it is queried with SQL.
- Typical format
- Structured tables with a defined schema
- On GCP
- BigQuery
- Question it serves
- "What is revenue per region per month?"
- Data engineer focus
- Core
A slice for one team
A subset of the warehouse shaped for a specific group, such as finance tables for finance or feature tables for data scientists. Usually owned by that department.
- Typical format
- Curated tables or views per audience
- On GCP
- BigQuery datasets · views
- Question it serves
- "Just show finance what finance needs."
- Data engineer focus
- Good to have
Data becomes information
The end of the pipe, where data goes back to humans (or models) as something useful. Three common uses: reporting, ad hoc questions, and machine learning.
- Typical format
- Dashboards · query results · trained models
- On GCP
- Looker Studio · BigQuery console · Vertex AI
- Question it serves
- "What should we do next quarter?"
- Data engineer focus
- Good to know
Underneath every stage sit two more jobs: an orchestrator that schedules and orders the data movement (Cloud Composer on GCP), and the infrastructure that runs it.
02
Warehouse vs lake
Both centralise data. The real difference is architecture. Flip the switch and watch what happens to the four building blocks.
One monolithic package. You put data in with SQL and read it back with SQL. Nothing inside can be swapped.
| Warehouse | Lake | |
|---|---|---|
| Building blocks | Bundled together | Separated, mix and match |
| Schema | Defined before loading | Optional, applied later |
| Access | Through SQL tables | Direct to files, any engine |
| Best at | Fast, governed analytics | Cheap, scalable raw storage |
Modern designs use both. On GCP that is typically Cloud Storage and Dataproc for the lake, and BigQuery for the warehouse.
03
ETL or ELT?
Only the order of two letters changes, but it decides where the heavy work happens. Decide where each situation should transform its data.
-
01 A 200 GB MySQL export needs to reach BigQuery, and analysts will shape it with SQL afterwards.
-
02 The target is a small reporting database that struggles with heavy joins and aggregations.
-
03 Customer names and phone numbers must be masked before the data leaves the source network.
-
04 IoT devices emit JSON constantly. You want to keep everything now and decide on a schema later.
04
What "big data" means in practice
Big is relative to your system. When a file is too large for one machine, a distributed file system splits it into chunks and spreads them over a cluster. Try it.
A metadata layer records which chunk lives on which machine, so the pieces can be reassembled. This demo spreads chunks evenly. Real systems also keep extra copies of each chunk, which is not modelled here.
05
Key terms
The vocabulary this chapter uses. Later chapters assume it.
- Upstream / downstream
- Where data comes from, and where it is going. Water in a pipe.
- Data silo
- Data held by one team or system that other teams cannot easily reach.
- Data lake
- Central store of raw files with storage, compute and schema decoupled.
- Data warehouse
- Modelled, queryable store (storage, compute, schema and SQL bundled).
- Data mart
- A warehouse slice shaped for one department or audience.
- ETL / ELT
- Extract-Transform-Load vs Extract-Load-Transform: where the transform runs.
- Cluster
- Multiple machines working together as one system.
- Metadata
- Data about data, such as which chunk of a file lives on which machine.
06
Check yourself
Pick an answer to see the reasoning straight away. Your answers are saved in this browser.
Q2
Which statement best captures the key technical difference between a lake and a warehouse?
Q3
Your target is BigQuery and your source is a 200 GB MySQL dump. Which pattern is the sensible default?
Q4
Is 50 GB of tweets "big data"?
Q5
A 1.15 GB file is stored in 128 MB chunks. About how many chunks, and what records where they live?
Study notes in my own words, following the structure of Adi Widjaja's Data Engineering with Google Cloud Platform (Packt, 2022). GCP changes quickly, so check each service's current documentation before relying on names, prices or console steps.