Data Ops
Hands-on guides to data tooling, infrastructure, and operations — scheduling, pipelines, access control, and the engineering behind data workflows.
- A quick walkthrough BigQuery Remote Functions
Explore how BigQuery Remote Functions enable data processing using external resources with step-by-step guidance
- Exploring the BigQuery Query History
Access BigQuery query history via the console's Job History or INFORMATION_SCHEMA.JOBS. Filter by user, time, and status to audit cost and execution details.
- Why you should care about VS Code Remote Development
Discover the benefits of VS Code Remote Development and enhance your workflow with efficient resource management using SSH and remote servers
- Importing Google Sheets into BigQuery
Step-by-step guide to importing a Google Sheets spreadsheet into BigQuery as an external table via Google Drive, including schema setup and viewing the...
- Linting BigQuery SQL with sqlfluff
How to install and use sqlfluff to lint and auto-fix BigQuery SQL code, configure dialect-specific rules, and integrate it as a pre-commit hook in your...
- Using GCP Cloud Functions in Data Engineering
How Google Cloud Functions fit into data engineering: serverless ETL, lightweight APIs, file parsing, and orchestration via Cloud Workflows — with key...
- Using Labels in BigQuery
BigQuery labels are key-value tags on jobs, tables, and datasets. Use them to break down billing by team, filter INFORMATION_SCHEMA, and manage via Terraform.
- Scheduled queries in BigQuery
Create BigQuery Scheduled Queries via the console, Terraform, or REST API. Covers cron syntax, write disposition, backfill, and monitoring execution history.
- Row-level access security in BigQuery
How to create and delete BigQuery row-level access policies so users only see the rows they are permitted to, with a working example using service...
- Loading data from Google Cloud Storage into BigQuery using Cloud Workflows
Step-by-step guide to loading CSV files from Google Cloud Storage into BigQuery using Cloud Workflows. Learn to set up IAM, auto-detect schemas, and load...