graphical representation

The big query demo found on qlikview seems very interesting. http://ap.demo.qlikview.com/QvAJAXZfc/opendoc.htm?document=qvdocs/American%20Birth%20Statistics.qvw# But I did not understand how do I...

BigQuery - Check if table already exists

I have a dataset in BigQuery. This dataset contains multiple tables. I am doing the following steps programmatically using the BigQuery API: Querying the tables in the dataset - Since my response...

Merging tables in Bigquery

I have 2 web sites and i am sending all the GA data into Big Query. Since Big Query would create different tables for each site, i would like to merge them into one. Also i would like to create a...

BigQuery: Create column of JSON datatype

I am trying to load json with the following schema into BigQuery: { key_a:value_a, key_b:{ key_c:value_c, key_d:value_d } key_e:{ key_f:value_f, key_g:value_g } } The keys under...

Need BigQuery SQL query to collect time on page from Google Analytics data

can anyone help with a BIgQuery SQL query to extract the time on page for a specific page from Google Analytics data please? For every visitorId who has visited a particular page I would like the...

BigQuery - How to query the number of columns in a table?

I want to query a table to get how many columns in that table and the name of each column. this post tells us how to do it in BQ command line interface, but can we do it using query?

BigQuery: PI() in Standard SQL

I've been using Google BigQuery's legacy SQL for a while and when I need the number Pi, there is a convenient PI() function: SELECT PI() But in Standard SQL, this function no longer exists. I...

how to get the most recent dataset using a wildcard in google bigquery

if I have a series of fact tables such as: fact-01012001 fact-01022001 fact-01032001 dim001 dim002 a wildcard will allow me to search all three, for example: select * from fact-* is there a...

Comparison of loading from different file formats in BigQuery

We currently load most of our data into BigQuery either via csv or directly via the streaming API. However, I was wondering if there were any benchmarks available (or maybe a Google engineer could...

Google Cloud Composer using google-cloud-bigquery python client library

I'm trying to run a DAG in Google Cloud Composer in which the first component is to use a http GET request to call an API and then use the python-client library to insert the json into a BigQuery...

How to take the incremental data backup of a table in Bigquery?

Please explain with an example of incremental backup in BigQuery. Example: I have a table name called TEST_TABLE. Day 1: full backup (means exported to Cloud storage): at 10 am EST I have...

How to set permissions for specific dataset on Google BigQuery?

I am trying to set permissions on BigQuery in order to have users being able to see and query tables on one dataset but being able to edit, create and delete tables on another dataset. I'm not...

Converting string column to timestamp in BigQuery?

I am trying to convert a column in a BQ table to timestamp. I have two string columns, one for utc_hour (string 0-23) and utc_day (string yyyymmdd) imported from a public data source. I merged the...

Listing all the partitions from BigQuery partitioned table with require_partition_filter

I am trying to find a way to list the partitions of a table created with require_partition_filter = true however I am not able to find the way yet. This is table creation script CREATE TABLE...

BigQuery Query working with multiple "likes" but not working with "in"

I would like to isolate some emails with specific titles. I can use multiple "like"s connected with an ORs in the where clause. This gives me a number of results. However, if I try to do a ____ in...

How to show and change user in Scheduled Queries

Some of the scheduled queries in Google Cloud Platform suddenly don't run anymore, with the message "Access Denied: ... User does not have bigquery.tables.get permission for table..." First, is it...

Bigquery export splits into multiple files with some empty files

I am trying to use the bigquery export functionality to push data out to GCS in json format. At the end of the process inorder to validate the count of exported records in the GCS file, I am...

Get environment variables in a cloud function

I have a Cloud Function in GCP that queries BigQuery in a specific project/environment. As I have multiple environments I would like to get the current project/environment of the cloud function....

Permission denied when using BigQuery public data set as a source

When setting up a source that points to a Big Query public data set and using the source() function to reference it in my dbt model, I get this error on dbt run: Database Error Access Denied:...

Google Data Studio - BigQuery Data Source List, how to keep track of of bigquery tables used as data source

For example, I use dozens BigQuery tables as the data sources for dozens Google Data Studio Dashboards. Some data sources of BigQuery tables got updated. for example, table_a1 etc. now it would be...

how to avoid [Missing close double quote (") character.] error in google bigquery

I am receiving the error below when I want to upload a data to google big query from a csv file. I have read many similar questions but still couldn't find a simple answer around how to fix...

what is the difference between BigQuery and Storage on GCP?

Hi guys I am using GCP for the first time and while I walking through the a project's cloud function example with the mock data, I got confused about similarities/differences of each one and I...

Possible to group by a user metadata label that includes dashes, in MQL?

I am trying to use MQL queries to export data from Google Cloud Monitoring to BigQuery for long-term archiving. I've found that queries that group-by user generated metadata labels including a...

Using apache-airflow-providers-snowflake on airflow (no module named Snowflake)

I have installed package apache-airflow-providers-snowflake on airflow on docker and i am getting error No module named Snowflake Please refer attachment (check the error mentioned for the...

Dataflow Bigquery-Bigquery pipeline executes on smaller data, but not the large production dataset

A little bit of a newbie to Dataflow here, but have succesfully created a pipleine that works well. The pipleine reads in a query from BigQuery, applies a ParDo (NLP fucntion) and then writes the...

Unrecognized name: employees at [9:8]

This is giving me an unrecognized name error. Why? SELECT employees.name AS employee_name, employees.role AS employee_role, departments.name AS department_name FROM ...

from Google import Create_Service ModuleNotFoundError: No module named 'Google'

I'm trying to use Gmail api in python to send email but I cant get past importing the Google module despite using "pip install --upgrade google-api-python-client" or "pip install google". However...

Loading dynamic schema JSON files into a BigQuery table using autodetect

This question is very similar to this one - but using the python API, and a couple of years later - and as the solution there is not complete, I thought it's best to open a new question. I have a...

Batch Update Bigquery results into google sheets(about 50k results)

I am trying to use Script editor(Apps script) to stream BigQuery results into google sheets. I'm following these [document][1] . The issue I have, the data doesn't load fully, it hangs. I have a...

Unable to determine type for field 'T_DATE'. warnings.warn("Unable to determine type for field '{}'.".format(bq_field.name))

I want to convert the biquery query result into dataframe using following code. Same code works file on google jupyterLab Notebook but throwing error on my local. from google.cloud import...