What are GCP Cloud Workflows and how can they help you as a Data Engineer

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
Search for a command to run...

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
No comments yet. Be the first to comment.
Short, practical posts on SQL and BigQuery — from core language features to advanced query patterns. A reference for data practitioners at every level.
Remember that in additional to the Cloud Console GUI or client libraries, you can also perform BigQuery tasks using the bq command-line interface. It's included with the gcloud SDK and once you have logged in, you can perform a number of operations s...
Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations

BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da

Query your data lake with warehouse-grade security and performance — without moving a single file.

Ever run a heavy BigQuery SQL query, processed gigabytes of data — and then accidentally closed the tab or forgot to save the results? 😬 Don't re-run it. Your results are still there. BigQuery automa

You can use query parameters in BigQuery hashtag#SQL (now in the console as well!) — but how are they different from variables, and when should you use each? Both parameters and variables act as place

A reminder that you don't necessarily need Airflow / Composer to orchestrate data in and out of BigQuery. If your workflows are not that complex or numerous, take a look at Google Cloud Workflows.
It's a serverless orchestration engine, with good integration with other GCP services, but you can also call external APIs.
Some advantages I see:
- serverless, no servers to manage
- your workflows stored as code, manage Infra as Code with Terraform
- interact with external APIs
- good connectors to interact with other GCP services
- straightforward pricing: at the time of writing: 5k steps free per month, 0.01 USD per 1k steps after, external API call are priced separately (2k/mo free, 0.025 USD per 1k after)
I've used it before to load data from Google Cloud Storage into BigQuery tables on a schedule and perform other simple tasks. It gets the job done with no fuss. You can of course schedule the Workflow to run with Cloud Scheduler.
See my previous post for a short walk through showcasing how BQ and Workflows play together.
Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.
Enjoyed this? Here are some related articles you might find useful: