Write better SQL queries faster using Mock Data

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
Search for a command to run...

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
No comments yet. Be the first to comment.
Short, practical posts on SQL and BigQuery — from core language features to advanced query patterns. A reference for data practitioners at every level.
Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations

BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da

Query your data lake with warehouse-grade security and performance — without moving a single file.

Ever run a heavy BigQuery SQL query, processed gigabytes of data — and then accidentally closed the tab or forgot to save the results? 😬 Don't re-run it. Your results are still there. BigQuery automa

You can use query parameters in BigQuery hashtag#SQL (now in the console as well!) — but how are they different from variables, and when should you use each? Both parameters and variables act as place

🛠️ Here's a technique I frequently employ to streamline my query development process.
Ever find yourself stuck in the complexity of writing non-trivial queries, especially with voluminous tables? 🤔 The amalgamation of query complexity and business problems can sometimes make writing your #SQL take more time than necessary.
When exploring a new coding approach, utilizing an unfamiliar function, or navigating through unseen data, I create a simplified, representative example using dummy data and a few test cases.
In a blank #BigQuery window, I write a couple of Common Table Expressions (CTEs) to simulate input, then focus on mimicking only the pivotal columns - maintaining the grain and incorporating one representative value, along with the join columns. A single value can often represent a surrogate key for the grain of the tables.
🚀 This approach enables me to:
- Iterate rapidly, considering edge cases irrespective of the data subset.
- Sharpen the technical requirements by applying the solution to the available data and addressing the business question.
- Contemplate how the technical solution can be optimized for data scale and manage real-world corner cases or unexpected scenarios.
- Save processing costs, since I'm working only on kilobytes of data until I have a solution that is >80% ready
👀 So, the next time you have to write a big query (pun unintended), especially when dealing with unfamiliar data or functions, take a moment to code a swift example in a blank BigQuery window. It pays off to think simply first.
🔄 Is this Test-Driven Development (TDD) for BigQuery? Perhaps. I like to term it Iterative Development.
Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.