GCP Data Engineering · Chapter 12

Boosting Your Confidence as a Data Engineer

You have now met most of the GCP data toolbox. This last chapter turns that into confidence: how the certification is structured, the services the book only mentioned, twelve exam-style scenarios in original wording, and a map to check what you can explain.

0 of – checks answered

The Professional Data Engineer exam

Certifications validate what you know, and preparing for them teaches you more than passing does. Google offers foundational, associate and professional levels; data engineering has one professional-level certification.

Level	Recommended for (per the book)
Foundational	Anyone with no hands-on Google Cloud experience
Associate	About 6 months or more of Google Cloud experience
Professional	About a year of Google Cloud experience and several years in the industry

Experience recommendations and exam contents change. Treat this table as a rough guide and read Google's current certification page.

What the exam covers

Pick the right architecture

Choosing storage, processing and messaging services for a set of requirements, and justifying the choice. Questions turn on keywords: "Spark", "streaming", "semi-structured", "quickly".

Book chapters: 1, 2, 3, 5, 6
Typical skill: Match requirements to services

The exam is broader than this course and asks detailed, edge-case questions, so prepare even if you have hands-on experience. Related certifications for expanding your range include Cloud Architect, DevOps Engineer and Machine Learning Engineer.

Services this course only touched

The exam expects you to know the positioning of a few more services. You need to know what each is for and how to choose, not every feature.

Which application database?

The book's decision tree is deliberately simple: first volume (roughly over or under 1 TB a day), then whether the data is structured. Try it.

Data volume Shape of the data

01 Structured order data for an application, under 1 TB a day, needing SQL.
02 Flexible product-catalog documents for a web app, modest volume, semi-structured.
03 Very high volume, structured data that needs global consistency and SQL.
04 Over 10 TB of sensor readings, NoSQL, with a real-time app that needs high throughput and low latency.

Real choices also weigh availability, replication (zonal, regional, global), price and team expertise, and these services keep evolving. Firestore now includes Datastore's role, and Spanner has widened. Learn positioning, then read current docs.

Bigtable in a nutshell

A managed wide-column NoSQL database, closely related to Apache HBase, with very high throughput, low latency and high availability. It suits time series, transaction and price histories and IoT, and is the wrong tool for warehouse-style joins. BigQuery can ingest streams but is not a low-latency serving database. Easy speed-ups: SSD instead of HDD, and above all a good row key.

01 Row key is the website domain name, so writes pile onto neighbouring rows.
02 Row key is the plain event timestamp.
03 Row key is the reversed domain name.
04 Row key is a device id followed by the timestamp.

Storage classes and getting data into GCP

Data is read about Data to move into GCP

Cloud Storage class
Transfer tool

Service	Positioning
Cloud Logging	Collects logs from all GCP services out of the box; can export to Cloud Storage, BigQuery or Pub/Sub
Data Loss Prevention	Detects and removes PII such as names, phone and card numbers, even inside free text, before loading
Dataprep	No-code, GUI data preparation: profile, clean and transform. Useful when nobody codes
Storage Transfer / Transfer Appliance	Bulk data migration into Cloud Storage; a physical appliance for very large offline transfers

Chapter map: can you explain it?

For each chapter, one question you should now be able to answer out loud. If you hesitate, revisit that chapter's interactive parts.

Chapter	You should be able to explain
1 Fundamentals	Why a lake and a warehouse differ, and when ETL beats ELT
2 GCP map	When to choose VM-based, managed or serverless; user versus service accounts
3 BigQuery	Append, truncate and empty loads; event versus snapshot data; facts and dimensions
4 Composer	What makes a task idempotent, and why sensors beat fixed time buffers
5 Dataproc	When an ephemeral cluster is wrong, and why RDDs are lazy
6 Streaming	Topic, subscription, subscriber and ack; why streams need windows
7 Looker Studio	How a dashboard multiplies BigQuery cost, and what materialized views do
8 ML	Which of four model approaches fits a situation; why pipeline steps are stateless
9 IAM	Where to grant a role to get the right inheritance, and what policy tags do
10 Cost	Machine, usage and flat pricing; how partitions and clusters cut bytes scanned
11 CI/CD	Why a failing test step stops a bad DAG reaching production

The road ahead

The book closes with a view of where data engineering is heading. It is an opinion, not a forecast, but a useful one to think against.

Then	Now	Next (author’s view)
Data modellers, DBAs and ETL developers on proprietary on-premises tools	A distinct, mature data engineer role built on shared principles and cloud services	The role splits into more specific ones, and more non-engineers write SQL
On-premises warehouses	Big data and the cloud are the norm, not the future	More cloud adoption in regulated industries; governance, security and multi-cloud grow
Engineers own all transformation	Engineers own extract, load and transformation	Engineers focus on the foundation: pipelines, security and governance

If you are new, start small. Many roles need only one area, such as BigQuery or Spark. Build capacity step by step, and when in doubt, return to the principles: least privilege, sound data modelling, and ETL versus ELT.

A study plan

Ticks are saved in this browser.

Read the current official exam guide
Sections, weightings and the services list change. Use Google’s own page as the source of truth, not any book.
Map the exam topics to what you have actually built
Mark topics you have done hands-on and topics you have only read about. Do the second group next.
Fill the services this book only mentions
Bigtable, Spanner, Firestore/Datastore, transfer services, Data Loss Prevention, Dataprep and logging. Learn each one’s positioning, not its every feature.
Do the quiz above cold, then reset and repeat after a week
Spaced repetition beats one long session. Note which chapter each miss belongs to.
Build one small end-to-end project of your own
Ingest, warehouse, orchestrate, govern, cost-check. A real project teaches judgement that exercises cannot.

Key terms

Bigtable: Managed wide-column NoSQL for huge, low-latency workloads.
Spanner: Globally scalable relational database.
Firestore / Datastore: Managed document NoSQL database.
Row key hotspot: A key design that sends most traffic to one node.
Storage classes: Standard, Nearline, Coldline and Archive: cheaper for colder data.
DLP: Data Loss Prevention: finds and masks sensitive data.
Overfitting: A model that fits training data too closely and fails on new data.
Regression: Predicting a numeric value from known inputs.

Review quiz: twelve scenarios

These cover the four exam sections in the exam's scenario style, written for this course. Pick an answer to see the reasoning.

A company has dozens of Spark jobs on its own servers and wants to move to GCP quickly with little rewriting. What is the natural fit?

Logs stream from thousands of devices and must be processed and stored for analytics. Which combination?

A mobile app needs a managed database for a 400 GB catalogue of semi-structured product documents. Which is the best fit?

Billing, shipping and analytics systems must each receive every order event independently. How should Pub/Sub be set up?

Every night at 05:00 you must extract from several Cloud SQL databases, land files in Cloud Storage, load BigQuery and run transformations. What should orchestrate it?

Data admins, data providers and data readers all use Cloud Storage and BigQuery. How should access be granted?

A web app must extract text from uploaded photos and score its sentiment, with no ML staff, within weeks. What approach?

You want to predict tomorrow’s electricity demand, a numeric value, from past data. Which type of model fits?

A model scores 99% on its training data and 60% on new data. What is the likely problem?

Q10

A BigQuery table holds two years of events, but most queries need only the last 10 days, and cost keeps rising. What helps most?

Q11

One node in a Bigtable cluster fails. What happens to the data?

Q12

A BI report on BigQuery shows numbers that look stale. What is the first thing to try?

Study notes in my own words, following the structure of Adi Widjaja's Data Engineering with Google Cloud Platform (Packt, 2022). GCP changes quickly, so check each service's current documentation before relying on names, prices or console steps.