Skip to main content
HashnodeDatawise — SQL, BigQuery & Python for Data EngineersDatawise — SQL, BigQuery & Python for Data Engineers

Command Palette

Search for a command to run...

pre_ and post_operations in Dataform

Published
•1 min read
pre_ and post_operations in Dataform
C
Constantin Lungu

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified

Part of seriesPractical SQL

Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations).

In practice it enables you to cleanly:

➡️ Declare and set a variable

➡️ Clean up before inserting into a table

➡️ Compute a watermark for your incremental model

➡️ Log run metadata after execution

➡️ Perform a maintenance task

What's the most interesting use case you've seen for pre_operations / post_operations (or pre_hook / post_hook if you're on dbt)?

#bigquery#sql#dataform#google-cloud

Comments

Join the discussion

No comments yet. Be the first to comment.

Practical SQL

Part 1 of 50

Short, practical posts on SQL and BigQuery — from core language features to advanced query patterns. A reference for data practitioners at every level.

Up next

BigQuery Object Tables: A Practical Introduction

BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da

More from this blog

D

Datawise — SQL, BigQuery & Python for Data Engineers

207 posts

Data Engineer with a passion for transforming complex data landscapes into insightful stories. Here on my blog, I share insights, challenges, and the ever-evolving dance of technology and business.