Using GCP Cloud Functions in Data Engineering

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
Search for a command to run...

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
No comments yet. Be the first to comment.
Hands-on guides to data tooling, infrastructure, and operations — scheduling, pipelines, access control, and the engineering behind data workflows.
SQL folks, are you using any linters? Myself, I've used linters such as Pylint for Python, but in the last 3 years, despite working with BigQuery daily, I didn't use one. First, what is a linter? If you haven't encountered the term before, a linter l...
Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations

BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da

Query your data lake with warehouse-grade security and performance — without moving a single file.

Ever run a heavy BigQuery SQL query, processed gigabytes of data — and then accidentally closed the tab or forgot to save the results? 😬 Don't re-run it. Your results are still there. BigQuery automa

You can use query parameters in BigQuery hashtag#SQL (now in the console as well!) — but how are they different from variables, and when should you use each? Both parameters and variables act as place

Some of the most interesting Data Engineering projects I've worked with leveraged Google Cloud Functions. It's a serverless execution environment that lets you run your code without provisioning or managing servers.
I found it to be very versatile, even more so for Data Processing tasks. Things like ingesting data, developing a lightweight API or consuming data from an endpoint, parsing and transforming a flat file.
As expected, it's very well integrated with other GCP services, so you could easily say, orchestrate run from a Google Cloud Workflow (a pairing that I used and enjoyed).
The advantages are pretty obvious: no server to manage, pay per invocation, scalability (if you need it) and simplicity to get started.
Now, there are of course a lot of things to think about when setting up such serverless functions :
- the size of the unit of work and runtime resources
- authentication (unauthenticated/authentication required)
- networking - where can the function can be invoked from
- auto-scaling and concurrency
- avoiding cold starts
- reusing heavy computations across invocations





But overall, it felt like the setup was pretty straightforward and the first time I tried it, I was able to get off the ground pretty quickly. You can of course test the function on your local machine (until you're happy with it) and automate its deployment with Terraform.
When used properly, a cloud function can be very useful for Real-Time and Batch ETL, automation and API Development, all while being scalable and flexible.
Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.
Enjoyed this? Here are some related articles you might find useful: