pre_ and post_operations in Dataform

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
Search for a command to run...

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
No comments yet. Be the first to comment.
Short, practical posts on SQL and BigQuery — from core language features to advanced query patterns. A reference for data practitioners at every level.
BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da
BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da

Query your data lake with warehouse-grade security and performance — without moving a single file.

Ever run a heavy BigQuery SQL query, processed gigabytes of data — and then accidentally closed the tab or forgot to save the results? 😬 Don't re-run it. Your results are still there. BigQuery automa

You can use query parameters in BigQuery hashtag#SQL (now in the console as well!) — but how are they different from variables, and when should you use each? Both parameters and variables act as place

Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations).
In practice it enables you to cleanly:
➡️ Declare and set a variable
➡️ Clean up before inserting into a table
➡️ Compute a watermark for your incremental model
➡️ Log run metadata after execution
➡️ Perform a maintenance task
What's the most interesting use case you've seen for pre_operations / post_operations (or pre_hook / post_hook if you're on dbt)?
