Skip to main content

Command Palette

Search for a command to run...

DELETE + INSERT vs MERGE in BigQuery

Updated
1 min read
DELETE + INSERT vs MERGE in BigQuery
C

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified

How do you merge changes from staging tables into target tables in BigQuery?

I've previously covered swapping out partitions using bq command and using constant false predicate "MERGE on FALSE", but I've learned that you can now DELETE entire partitions for free (provided a filter on the partitioned column is used) from tables.

That means that instead of merging your changes the old-fashioned way, it might be well worth DELETING the days you would like to update and INSERTING the entire days data back sourced from the staging table.

Here's a comparison of the two approaches for the same source and destination tables. As you can see the amount of processed data can be wildly different between the two.

Happy querying!

Found it useful? Check out to my Analytics newsletter at notjustsql.com.

Practical SQL

Part 1 of 50

Short, practical posts on SQL and BigQuery — from core language features to advanced query patterns. A reference for data practitioners at every level.

More from this blog

D

Datawise — SQL, BigQuery & Python for Data Engineers

205 posts

Data Engineer with a passion for transforming complex data landscapes into insightful stories. Here on my blog, I share insights, challenges, and the ever-evolving dance of technology and business.