pre_ and post_operations in Dataform

Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations).

In practice it enables you to cleanly:

➡️ Declare and set a variable

➡️ Clean up before inserting into a table

➡️ Compute a watermark for your incremental model

➡️ Log run metadata after execution

➡️ Perform a maintenance task

What's the most interesting use case you've seen for pre_operations / post_operations (or pre_hook / post_hook if you're on dbt)?

config {
  type: "incremental",
  uniqueKey: ["order_id"]
}

pre_operations {
  DECLARE max_date DATE
  DEFAULT (
    SELECT COALESCE(MAX(order_date), DATE('2000-01-01'))
    FROM ${self()}
  );
}

SELECT *
FROM ${ref("orders")}
WHERE order_date > max_date

post_operations {

  -- Track every run in your audit log
  INSERT INTO ${ref("pipeline_audit")} (table_name, loaded_at)
  VALUES ("${self()}", CURRENT_TIMESTAMP());
}