Today I'd like to talk about a special variety of external table: the BigLake table. It's built to bridge data lakes and data warehouses.

BigQuery External Tables: Limitations and Pain Points

With a regular external table, users need access both to the BigQuery table and to the underlying external data source.

If the data is in Cloud Storage, a user needs BigQuery permissions, but also permissions on the bucket and objects. This might be fine for a quick exercise, but the pain points are real:

• you manage permissions in multiple places, for different types of resources

• bucket-level access can be too broad

• it's harder to apply table-like governance on files

How BigLake Tables Work: Access Delegation Explained

BigLake tables use a BigQuery Connection that accesses the files on behalf of the users. Users don't need direct access to the underlying buckets.

This enables table-level security on external data, including:

• row-level security

• column-level security

• dynamic data masking (for Cloud Storage BigLake tables)

BigLake also enables — powered by BQ Omni — reading data from Amazon S3 and Azure Blob Storage.

Metadata Caching in BigLake Tables: Faster Queries, Better Plans

When querying external data, BigQuery needs to inspect the files first: what files exist, how they are partitioned, and what metadata they contain. With classic external tables, every query triggers a listing operation against the underlying storage.

If you have a small number of files, this is barely noticeable. If you have thousands or millions of files, especially Hive-partitioned data, it becomes painful.

BigLake tables unlock metadata caching. With it enabled, BigQuery skips the listing on every query and prunes files and partitions faster — avoiding reading unneeded files altogether.

For Parquet BigLake tables, metadata caching also collects table statistics, which helps the optimizer produce better query plans.

The cache has a staleness window you control, and you choose between automatic or manual refreshes (the manual option runs a stored procedure, useful if you want to make it event-driven).

Creating a BigLake Table: Step-by-Step SQL Example

Say we have some sales data in Google Cloud Storage as follows

It is Hive partitioned Date -> Country.

In order to create a BigLake table we need to do the following:

Create a BQ Connection

Now, we need to grant this service account access to the GCS bucket

We can now create the BigLake table:

CREATE EXTERNAL TABLE `learning.orders_biglake`
(
  order_id STRING,
  customer_id INT64,
  channel STRING,
  amount NUMERIC,
  discount_amount NUMERIC,
  created_at TIMESTAMP
)
WITH PARTITION COLUMNS
(
  order_date DATE,
  country STRING
)
WITH CONNECTION `projects/your-gcp-project/locations/eu/connections/demo-biglake-connection`
OPTIONS (
  format = 'CSV',
  skip_leading_rows = 1,
  field_delimiter = ',',
  hive_partition_uri_prefix = 'gs://datawise-biglake-hive-demo-bucket/orders',
  uris = ['gs://datawise-biglake-hive-demo-bucket/orders/*'],
  max_staleness = INTERVAL 1 DAY,
  metadata_cache_mode = 'AUTOMATIC'
);

A few things to note:

• max_staleness — how old the cache can be before BigQuery goes back to storage. Minimum is 15 minutes

• metadata_cache_mode — AUTOMATIC refreshes on its own; MANUAL lets you trigger it via stored procedure, useful for event-driven pipelines

• consider adding require_partition_filter to force callers to filter on a partition key and avoid full file scans

The table is now created and can be queried like any other BigQuery table.

Before you go

A couple of things worth knowing:

• an existing classic external table can be upgraded to a BigLake table without recreating it from scratch

• metadata cache refreshes incur processing costs — worth keeping in mind if you're dealing with a large number of files

Don't re-run it. Your results are still there.

BigQuery automatically saves query results to an anonymous temporary table for 24 hours. You can find it under Job Information → Temporary Table.

This is also the engine behind BigQuery's caching behavior: if you run the exact same query again within that window — with no changes — BigQuery serves results directly from cache, at no charge.

One caveat worth knowing: if your result set exceeds 10 GB (that's the output, not the data scanned), it won't be cached. So for very large result sets, you'll want to write results to a permanent table explicitly.

Both parameters and variables act as placeholders and have a defined data type. The difference is where their value comes from and how they’re used.

Parameters (like @corpus)
👉 Are not computed inside the query
👉 Are passed from the outside (Python, UI, API, etc.)

Variables (DECLARE, SET)
👉 Are defined and computed inside a SQL script or stored procedure
👉 Let you store a value and reuse it later in the same script

So what’s the real difference?
➡️ Variables are essential for Dynamic SQL (EXECUTE IMMEDIATE)
➡️ Parameters can filter data, but cannot control identifiers (e.g. table or column names)

🚨 Security
When values come from user input or external sources, parameters are the safer choice—they reduce the risk of SQL injection.

🚅 Performance
Parameters may allow the optimizer to reuse execution plans, while variables can sometimes prevent that.

Found it useful? Check out to my Analytics newsletter at notjustsql.com.

It allows us to flatten JSON arrays and return a single flat ARRAY, no matter how many nested levels there are.

So where is this actually useful?
➡️ Handling heterogeneous JSON where the nesting depth isn’t consistent
➡️ Cleaning up malformed or jagged arrays
➡️ Normalizing data before UNNEST so you don’t get arrays of arrays

Where I would not use it?

👉 Don’t use it when the hierarchy matters. Flattening removes structural context, so you lose information about where an element came from.

Found it useful? Check out to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• The JSON datatype in BigQuery
• JSON datatype vs JSON-like STRING in BigQuery
• Extracting keys from JSON in BigQuery
• LAX JSON conversion functions in BigQuery

This is why, if you work with STRUCTs in SQL and try to find the latest non-empty struct using LAST_VALUE(...) IGNORE NULLS, you’ll notice it doesn’t help — because the struct, even when all fields are null, is still considered non-null.

LAST_VALUE only skips rows where the entire expression itself is NULL.

To fix this, we can adjust the logic in one of the following ways:
➡️ Setting the value to NULL when all fields are NULL
➡️ Using TO_JSON_STRING + NULLIF to treat such entries as “null”
➡️ Using REGEXP_CONTAINS (thanks ChatGPT) for more dynamic checks

Alternatively, we can just apply LAST_VALUE separately to each individual field in the struct.

If you're new to STRUCTs, see one of my previous posts.

Found it useful? Check out to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• Beware of ROW_NUMBER without ORDER BY
• Tidying up WINDOW functions in BigQuery with named windows
• Using RANGE in Window Functions in BigQuery
• Computing a cumulative sum in BigQuery

I needed a quick way to validate my assumptions about the table grain, identify contradicting observations (rows), and check for duplicates at the same time.

Therefore I decided to use a combination of TO_JSON_STRING and FARM_FINGERPRINT. The first creates a JSON representation of the entire row (given a table alias), while the second converts the resulting string into a INT64 hash.

By comparing the total number of rows in a group against the distinct count of these fingerprints, we can determine whether the proposed grain is correct and whether there are duplicates in the data.

This was a quick exercise but use this with care. Depending on your SQL implementation, data volumes and context, results may vary.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• 9 tips on writing cleaner SQL
• Order of precedence in SQL: WHERE vs HAVING
• Easy with that SELECT DISTINCT!
• Why you should use parentheses with AND & OR in SQL

Not to be confused with the WITH clause, which we use to define common table expressions (CTEs).

👉 What does a WITH expression do?
It lets you define a series of variables, scoped to a single expression. Each variable can reference previously defined ones (and table columns), and in the end, the whole expression returns a result.

📌 Where could this be useful?
Think back to the time before QUALIFY was supported. We often had to create an extra CTE just to filter with WHERE rn = 1 or for similar windowed calculations. When QUALIFY came, it saved us a bunch of boilerplate CTEs.
Well, WITH expressions have the potential to help in the same way — but for non-window calculations.

When working with complex formulas, you can’t reference (within the same SELECT) a column you just defined. The usual workaround is to push it into another CTE — which works, but feels verbose. I still opted to do it since it's important that the code stayed readable and maintainable.
Now WITH expressions give us a cleaner option and help avoid those 7-operand expressions. I, for one, plan on trying them out ASAP.

Has anyone here used them already? Any thoughts? Docs here.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

I recently faced a mystery that pushed me to the edge of despair.

It seemed like a simple issue at first glance. A report kept changing completely at random.

I spent a good few days chasing it across multiple weeks . Imagine juggling several tables with time travel, all joined together. Trying to catch the bug.

I started to wonder if time travel even worked correctly. I wasn’t able to reproduce previous states, even when all the inputs had data from that exact point in time.

I began to question if I’d make it. The culprit?

Take this as a cautionary tale against using ROW_NUMBER() OVER(PARTITION BY …) without an accompanying ORDER BY.

Tucked into a table somewhere, it haunted me and wreaked havoc. I don’t know if there’s a real use case for it like that — but expect surprises.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• Tidying up WINDOW functions in BigQuery with named windows
• Using RANGE in Window Functions in BigQuery
• Computing a cumulative sum in BigQuery
• Rolling period calculation in BigQuery

Say you have an input SCD2-style table with [valid_from, valid_to) and key-value attributes. Now you want to determine which attributes were valid at the same time for a given grain (e.g. id).

To do this, we:

1️⃣ Build an anchor table of all change points (start and end dates), grouped by id.

2️⃣ Generate date ranges using LEAD() over the change points, so we know the next boundary.

3️⃣ Join back to the original table to find which rows were active within each [valid_from, valid_to) segment.

4️⃣ Aggregate the key-value pairs as ARRAY<STRUCT<key, value>> to preserve temporal context.

We can now see that, for example, for the period between [2023-01-05, 2023-01-08), for id = 2, B was true and A was false.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• Joining temporal tables in BigQuery
• Generating a compact temporal table in BigQuery
• Compacting date intervals in BigQuery
• Transforming cumulative sums into monthly values

With that bold statement out there, here’s a BigQuery lesson drawn from real-world experience.

Imagine an SCD-2 table with tens of millions of rows containing product data.

For particular subintervals (even individual days) of the [valid_from, valid_to) period, we need to set some attributes (think: marking that the product is on sale, or is getting ready to be discontinued).

The simple and straightforward solution here would be to unpack the intervals with something like UNNEST(GENERATE_DATE_ARRAY(valid_from, valid_to)). Joining with a date dimension would work in a similar fashion.

This means that it is easy to now look up these attributes at (grain + valid_date).

But now we could be talking about several billion rows. Which you might join with other big tables.

The issues with this approach are quite obvious. Say we have a two-year interval. Unpacking this period would mean having 730 rows instead of one, while a majority of them would just be repeated data — there’s no actual change, just some spot changes for individual days or subintervals.

We can of course go ahead and compact this data back using an algorithm like the one I presented in one of my previous posts, but regardless of whether we do that or not, the whole processing would be incredibly expensive and inefficient.

A better approach: work with intervals

So, is there a solution here? Sometimes there might be.

Say for the two-year interval [2021-01-01, 2023-01-01], we’d need to mark the subperiod [2021-04-15, 2021-05-15] as the period when the product is on sale.

This data can be accurately expressed as the following intervals:

[2021-01-01 → 2021-04-15),

[2021-04-15 → 2021-05-15),

[2021-05-15 → 2023-01-01).

So in fact, we’re splitting the intervals into smaller subintervals based on what was true at each moment. Instead of unpacking and getting hundreds of records, we need only 3. I’ve illustrated this approach in another previous post.

Such a difference in cardinality has deep performance implications, allowing us to process data way more efficiently.

While this might be an unusual corner case, my initial opinion stands — if you can get away without UNNESTing your array or unpacking a date interval, you’ll manage to keep cardinality in check and save a lot of processing power.

If there’s no way around it for your particular case, at the very least, materialize the result (I know, it’s going to be a lot of storage) and tailor the partitioning and clustering according to the querying (especially joining with other big+ tables) downstream.

Enjoyed this? Here are some related articles you might find useful:

• Generating a compact temporal table in BigQuery
• Aggregating Multiple SCD-2 Attribute Timelines in BigQuery
• Using ARRAY_AGG in BigQuery
• Short, almost non-technical guide to SQL query tuning as a Data Engineer

While the performance gain when using these unenforced PK/FK constraints in general may be up for discussion, it's definitely nice to be able to see this table metadata there, including the table grain 👍

For a refresher on what these constraints are, see my previous post.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• Row-level access security in BigQuery
• Using subqueries with Row Level Security in BigQuery
• Why basic roles in BigQuery are a bad idea

Problem statement: compacting a SCD-2 table, essentially finding intervals that can be safely merged, turning two adjacent intervals with the same data into a single, bigger interval.

This particular input data guarantees these intervals cannot overlap (at the same grain), but there can be gaps. We're also talking about [left-inclusive, right-exclusive) intervals.

Here's a breakdown of how it all works:

1️⃣ we're starting by computing a hash of all the column of interest, excluding the grain (in my example: flag_a, flag_b)
2️⃣ then we use LAG() over grain window to detect whether the current row starts right after the previous one and if the hashes (so the 'payload' of the two rows) match
3️⃣ we mark the start of a new "segment" when either:
- attributes have changed (flags differ), so hash being different
- intervals are not adjacent (there are gaps)

4️⃣ use a cumulative SUM() over grain window to group rows into segment IDs

5️⃣ collapse each segment using MIN(valid_from) and MAX(valid_to)

We can now see that in our example that several intervals were merged into bigger ones.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• Joining temporal tables in BigQuery
• Generating a compact temporal table in BigQuery
• Aggregating Multiple SCD-2 Attribute Timelines in BigQuery
• Transforming cumulative sums into monthly values

I'm talking about the NULL-safe operators IS DISTINCT FROM and IS NOT DISTINCT FROM. I found about their existence from a Linkedin post.

Works on BigQuery too, so I guess less need of adding IFNULLs / COALESCE for safety.

PS This choice of keyword "FROM", together with the one in EXTRACT(HOUR FROM DATETIME '2021-01-01'), feels pretty weird.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• A couple of fun things about NULL in SQL
• Not all NULLS are the same
• COALESCE vs IFNULL vs NULLIF in BigQuery
• Controlling ordering of NULL values in the ORDER BY clause

Let's look at an example.

The dataset provides cumulative sales per fiscal year (July 1st - June 30th in this case). Our goal is to determine the actual sales for each month.

How do we do it?

1. Identify the fiscal year each period belongs to. We can use a UDF (as shown) or retrieve this from a date dimension table.

2. Use the LAG window function to retrieve the previous cumulative value (partitioned by our grain + fiscal year and ordered by period).

3. Subtract the previous cumulative value from the current one to derive the actual monthly sales.

• For the first month of a fiscal year, there’s no previous value, so we default to 0 in case of a NULL there.

Things to watch out for:
➡️ Gaps in the data: How do they impact the calculation? Are we okay with that?
➡️ Grain considerations: Do we need to do this per department? Per country? If so, adjust the PARTITION BY accordingly.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• Joining temporal tables in BigQuery
• Generating a compact temporal table in BigQuery
• Aggregating Multiple SCD-2 Attribute Timelines in BigQuery
• Compacting date intervals in BigQuery

This is especially useful when SQL lacks built-in support for your specific needs, and writing a UDF isn’t an option (for example, if you need a highly specialized Python function).

Before building your own, check out bigfunctions—many common use cases have already been solved by others!

What are remote functions in BigQuery?

These are a special type of function that delegates processing of input to an external resource, allowing us to:

• send data from BigQuery to Google Cloud Functions or other external services

• process it using a programming language

• return results to our query

Why is that important?

A Google Remote function can encapsulate any kind of logic in major programming languages.

This opens the door to vast a ecosystem of libraries such at the Python packages.

Let’s look at a step-by-step example of creating a Remote Function.

Step 1: Create the Cloud Run Function

Step 2: Create a connection

Step 3: Set up permissions

Step 4: Bind the connection with Cloud Run function

Test run

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Now, the Change History features in BigQuery sound pretty interesting.

BigQuery SQL has had the APPENDS table-valued function (TVF) for some time now, which works well for append-only scenarios. But it didn’t capture updates or deletes.

A few months ago a CHANGES TVF was added, which provides visibility into UPDATE and DELETE operations.

Unlike APPENDS (which works right out of the box), you need to enable change history tracking manually either at table creation or with an ALTER TABLE ... SET OPTIONS() command.

To illustrate how it all works I've:
1️⃣ Created a table 2️⃣ Inserted a row 3️⃣ Updated a row

As you will be able to see:
✅ APPENDS captures new rows only.
✅ CHANGES logs updates too (as a DELETE + INSERT).

Key things to note:

⚠️ Both features are still in preview, so not production-ready.
💰 Querying this data still incurs processing costs.
⏳ CHANGES only tracks modifications older than 10 minutes.
📦 Enabling Change History means extra storage costs for metadata.

Has anyone tried using these in real-life scenarios?

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Beyond the standard built-in functions like CURRENT_TIMESTAMP() or LENGTH(), BigQuery helps users to define custom functions that extend SQL capabilities. These user-defined functions belong to the larger category of routines (alongside stored procedures), enabling logic reuse.

🔹 Types of Functions in BigQuery

By Duration

➡️ Persistent functions – Stored in your dataset and reusable across all sessions

➡️ Temporary functions – Available only within your current session (created with TEMP keyword)

By Return Type

➡️ Scalar functions – Return a single value per input row (which can be complex types like structs or arrays) → typically used in SELECT or WHERE clauses

➡️ Table-Valued Functions (TVFs) – Return entire tables, requiring you to SELECT FROM the function

BigQuery functions can be written in either SQL or JavaScript.

Based on their processing nature:

➡️ Regular UDFs – Process individual rows, transforming inputs into a single output value

➡️ User-Defined Aggregate Functions (UDAFs) – Combine multiple rows into a single result using custom logic (currently in preview)

🔹 Beyond BigQuery: Remote Functions

For complex processing requirements, BigQuery offers remote functions, which allows us to:

➡️ Send data to Google Cloud Functions or other external services

➡️ Process it using a programming language

➡️ Return results to our query

This opens access to the vast ecosystem of libraries in languages like Python.

Their Place in Modern SQL

Back in the day when I just started with SQL Server, I used scalar functions sparingly (as a junior I was always warned about performance 🤓) and occasionally employed TVFs for small reusable datasets.

Today, with modern transformation frameworks like dbt and Dataform, I find myself almost not using BigQuery — the same reusable logic is now defined as macros or custom JS functions within these frameworks.

💡 I'm curious:

➡️ How often do you use UDFs or TVFs in your SQL environment?

➡️ Do you prefer handling reusable logic in your SQL code or in external frameworks?

➡️ Any interesting use cases you've seen for remote functions for unusual/specialized processing needs?

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

I initially experimented with a CASE WHEN in the join condition, hoping it would short-circuit, picking the first matching condition—just like in a SELECT clause. However, in a join, it evaluates all scenarios, so that didn’t work as expected.

But remember correlated subqueries? A correlated subquery runs once per row and can be embedded in the SELECT or WHERE clause. Essentially, it lets you create a dynamic query within a single data cell, based on the current row’s context. Check out this quick intro.

To avoid multiple joins, you can use a correlated subquery to fetch all possible combinations (previously handled by join conditions) and apply the same logic with ORDER BY and LIMIT to return exactly one value.

A word of caution: correlated subqueries execute once per row, which can impact performance, especially with large datasets. However, they’re a valuable tool in your SQL tool belt, particularly when other elegant solutions aren’t available.

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• Self-joins in SQL
• Anti-joins in SQL
• SEMI-JOINS in SQL
• NON-EQUI joins in SQL

It felt like that for me remembering about ROLLUP a few days ago.

I wrote about GROUP BY ROLLUP roughly 1.5 years ago—one of my first posts here. At the time, it was unfamiliar to me, and I had no idea I’d ever need it. But this week, I finally encountered a real use case.

𝐓𝐡𝐞 𝐏𝐫𝐨𝐛𝐥𝐞𝐦

Imagine we have sales data for a retail store, where each product belongs to a subcategory and a category (e.g., Apples → Fruits → Food).

We want to compute the average ordered quantity per product, but with a hierarchical fallback:

1. If there’s no product-level data, use the subcategory average.
2. If that’s missing, use the category average.
3. If still unavailable, fall back to the overall average across all products.

𝐇𝐨𝐰 𝐑𝐎𝐋𝐋𝐔𝐏 𝐇𝐞𝐥𝐩𝐬

When we GROUP BY ROLLUP (category, subcategory, product_id), we get multiple aggregation levels in one query:
✅ Per product
✅ Per subcategory
✅ Per category
✅ Across all rows

This allows us to build a lookup table, which we can use with multiple LEFT JOINs to apply the fallback logic.

𝐋𝐞𝐭'𝐬 𝐭𝐞𝐬𝐭 𝐢𝐭

Here’s how it works in practice:
• Apples → Direct sales data → AVG(quantity) = 6
• Mangoes → No past sales → Uses Fruits subcategory → AVG(quantity) = 4.67
• Cucumbers → No past sales, no Vegetables subcategory data → Uses Food category → AVG(quantity) = 4.67
• Washing Machine → No sales data, no relevant category → Uses overall average → AVG(quantity) = 6

𝐈𝐧 𝐥𝐢𝐞𝐮 𝐨𝐟 𝐚 𝐜𝐨𝐧𝐜𝐥𝐮𝐬𝐢𝐨𝐧

This was a fun experiment, but let’s be honest—this could also be done with window functions!

Still, ROLLUP provides an perspective, and I’m on the lookout for an even better use case.

Have you ever had an SQL feature suddenly “click” for you?

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

🚨 Turns out, I was wrong.

This was a great reminder of why understanding SQL’s order of execution is crucial!

I expected SUM(SUM(val)) OVER (PARTITION BY id) to accumulate incorrectly, but SQL’s execution order ensures that:

1️⃣ The GROUP BY clause first aggregates SUM(val) at the id grain.

2️⃣ Then, the window function is applied to the grouped result—not the raw data. Since there’s only one row per id, the window function correctly returns the expected value.

SQL doesn’t “re-sum” the window function like I feared. Instead, it partitions over the already-aggregated values—exactly as it should.

🔍 Have you ever misjudged a query’s behavior?

Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.

Enjoyed this? Here are some related articles you might find useful:

• 9 tips on writing cleaner SQL
• Order of precedence in SQL: WHERE vs HAVING
• Easy with that SELECT DISTINCT!
• Why you should use parentheses with AND & OR in SQL