Optimizing storage costs in BigQuery

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
Search for a command to run...

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
No comments yet. Be the first to comment.
Practical guides to making BigQuery queries faster and cheaper — partitioning, clustering, search indexes, time travel, cost optimization, and query tuning strategies.
When it comes to performance improvements and cost savings, handling only as much data as we need is very important. And partitioning is a cornerstone here. Now, when working with tables that are partitioned, BigQuery tries to exclude the partitions ...
Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations

BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da

Query your data lake with warehouse-grade security and performance — without moving a single file.

Ever run a heavy BigQuery SQL query, processed gigabytes of data — and then accidentally closed the tab or forgot to save the results? 😬 Don't re-run it. Your results are still there. BigQuery automa

You can use query parameters in BigQuery hashtag#SQL (now in the console as well!) — but how are they different from variables, and when should you use each? Both parameters and variables act as place

As promised on a previous post about optimizing costs for compute, I've promised a post about storage as well, so let's dive in.
So how does BigQuery charge for storage?
Two billing models - set at dataset level: logical and physical.
Logical table size is the default option, representing uncompressed size. It is cheaper per Gib (roughly half of physical) and you get time travel storage for free.
Physical storage represents compressed, actual size of physical bytes stored on disk. It's twice as expensive as logical and you need to pay for the time travel storage as well BUT if you have data that compresses well (for example by using arrays), you might be able to save storage costs.
Now, we can further split these down into:
- active storage: any table or partition (for partitioned tables) modified in the last 90 days
- long-term storage: a table or partition (for partitioned tables) that hasn't changed in the last 90 days, and is 50% cheaper than the active storage.
Here's what you can do to help bring these costs down:
- Identify what is taking up storage space: Look at information schema views such as INFORMATION_SCHEMA.TABLE_STORAGE to find out what is taking up the space that you pay for.
- Cleanup: Clean up unneeded data or archive in cheap long-term storage. Think about storing aggregated version of historical data and save on storage.
- Leverage table clones and snapshots in BigQuery to save on storing redundant data
- Shorten time travel window (which is 7 days by default) - if you would not need to restore this data or not for the entire 7 days, you can reduce this (dataset level) and store less data. A staging table that is recreated every time, for example, might not need the 7 days. Read more in the post about BigQuery time travel.
- Leverage long-term vs short-term data: any table or partition that you don't modify for 90 days becomes 50% cheaper to store, so maybe don't rebuild massive historical tables every time you run the process.
- Use the table and partition expiration settings: you can set particular tables or partitions to expire after a given time, so you don't store data that you don't need
Hope these tips are useful!
Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.
Enjoyed this? Here are some related articles you might find useful: