Search Indexes in BigQuery

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
Search for a command to run...

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
No comments yet. Be the first to comment.
Practical guides to making BigQuery queries faster and cheaper — partitioning, clustering, search indexes, time travel, cost optimization, and query tuning strategies.
Riding on the back of recent news that BigQuery table partition limit has just increased from 4k to 10k partitions, I wanted to talk a bit about the PARTITIONS view in INFORMATION_SCHEMA. I've previously posted about information schema views, but thi...
Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations

BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da

Query your data lake with warehouse-grade security and performance — without moving a single file.

Ever run a heavy BigQuery SQL query, processed gigabytes of data — and then accidentally closed the tab or forgot to save the results? 😬 Don't re-run it. Your results are still there. BigQuery automa

You can use query parameters in BigQuery hashtag#SQL (now in the console as well!) — but how are they different from variables, and when should you use each? Both parameters and variables act as place

Here's something that might be interesting if you analyze large volumes of STRING or JSON data in BigQuery.
Let's look at SEARCH indexes and what you need to know to get started.
You have a big table (>10 GB) with STRING or JSON columns which you perform text analysis on.
Search indexes can help retrieving data more efficiently from unstructured/semi-structured data - columns of type STRING, JSON, ARRAY of STRING, STRUCTS with STRING or JSON columns.
It can help optimize the usage of SEARCH funciton as well as other operators you use with string fields like 'STARTS_WITH', 'IN', '=' or 'LIKE'.
CREATE SEARCH INDEX term_search_index ON dataset.table_name(ALL COLUMNS/1 or more columns);
Based on what columns you've indexed, you can leverage the search index to search the entire table (columns with the compatible datatypes) or just a subset of columns of interest.
Check the 'Job Information' of your BigQuery. This will tell if you if an index was used, and if not, what was the reason.
Check out text analyzer options to see what different use case you can cover better. Maybe a future post about this 😁
- works best when you have a lot of distinct values (high query selectivity)
- if you've indexed all the columns any new compatible (STRING, JSON) column in that table will be indexed as well
Found it useful? Check out to my Analytics newsletter at notjustsql.com.