detectClassPathResourcesToStage - Unable to convert url

When I run the jar in the GCE, it had the following error: java -jar mySimple.jar --project=myProjcet Aug 13, 2015 1:22:26 AM com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner...

Why does Dataflow show Protobuf error messages on any Datastore error?

I have a simple Dataflow that copies from one Datastore to another. It generally succeeded, but on Kind __BlobInfo__ fails as follows. The entities in __BlobInfo__ are of course an essential part...

Error in Dataflow: io.grpc.StatusRuntimeException: UNAVAILABLE

I'm trying to build a Dataflow job that reads from BigQuery and sends messages to Pubsub topics. I've been struggling with inter-dependencies, and this is the latest issue. The Dataflow job...

Apache Beam Google Datastore ReadFromDatastore entity protobuf

I am trying to use apache beam's google datastore api to ReadFromDatastore p = beam.Pipeline(options=options) (p | 'Read from Datastore' >> ReadFromDatastore(gcloud_options.project, query) |...

Google Cloud Dataflow DatastoreIO Dependency Problems for Read and Write

What I need: the right combination of Dependency-Versions to be able to Read and Write from/to Datastore in Dataflow (v. 1.9.0) via DatastoreIO.v1().read/write and which dependencies need to be...

Importing translate from google.cloud

I am trying to do translate a document with google translate from the package google.cloud I already did: pip install google.cloud and the result was: Collecting google.cloud Using cached...

import apache_beam metaclass conflict

When I try to import apache beam I get the following error. >>> import apache_beam Traceback (most recent call last): File "<stdin>", line 1, in <module> File...

gRPC/proto Google cloud client libraries vs GAPIC Google cloud client libraries

I've mostly found working with google's client libraries easy to work with, intuitive, and well suited to idiomatic python, with the notable exception of auth (there is a special place in hell for...

ModuleNotFoundError using Google Cloud Speech

Edited: I installed Google Cloud Speech by using the followning command: pip install --upgrade google-cloud-speech when I run Python Program to convert Audio into Text using Python3, I get an...

Error in installing google-cloud

I am get some errors while trying to install google-cloud by using pip. I tried the following commands to install google-cloud: sudo pip install --upgrade google-cloud sudo pip3 install --upgrade...

ClassNotFoundException on CloudResourceManager$Builder

In a spring boot application running google cloud dataflow code. The dataflow takes data from google PubSub, transform incoming data and output result to bigquery for storage. The code does not...

CloudDataflow can not use "google.cloud.datastore" package?

I want to put datastore with transaction on CloudDataflow. So, I wrote below. def exe_dataflow(): .... from google.cloud import datastore # call from pipeline def ds_test(content): ...

error while importing WriteToDatastore (Apache Beam/Google DataFlow)

I'm trying to use an Apache Beam pipeline to write entities to the Google Cloud Datastore. For testing, I'm doing this in a local Python 2.7 virtual environment that is setup using the Apache Beam...

How to get the result of an SQL query from Big Query in Airflow?

Using Airflow I want to get the result of an SQL Query fomratted as a pandas DataFrame. def get_my_query(*args, **kwargs) bq_hook = BigQueryHook(bigquery_conn_id='my_connection_id',...

ImportError phonenumbers with google cloud dataflow python

I am trying to do a relatively simple import of the module phonenumbers in Python. I have tested the module on a seperate python file without any other imports and it works completely fine. These...

Unable to install pandas from pip2 from Homebrew

I install python2 in my Mac from homebre with brew install python2 And now I want to install pandas with pip2 pip2 install pandas But I'm receiving this error: grpc-google-iam-v1 0.11.1 has...

pip installing gsutil causes too many incompatible dependencies

When I run pip install -U pip, I get the following error: proto-google-cloud-spanner-v1 0.15.3 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is...

Python imp not finding modules

I'm trying to get the Google cloud modules for python working, but am having difficulty in imp seeing the modules correctly for verification purposes. I've even tried a few variants in spelling,...

Can no longer install `google-cloud-datastore` dependency for Google Cloud Dataflow jobs in requirements.txt file

Something similar to this last post happened again: https://stackoverflow.com/questions/50729803/google-python-cloud-dataflow-instances-broke-without-new-deployment-failed-pubs Essentially,...

How to deploy GCP App Engine worker?

I am trying to make personal apps on GCP- following the deploying a complex Python app to flexible Google Cloud environment tutorial [ https://cloud.google.com/python/getting-started/using-pub-sub...

Google Cloud Dataflow (Python) - Not installing dependencies correctly

I'm trying to run the official Dataflow example here:https://github.com/GoogleCloudPlatform/dataflow-prediction-example However, the Dataflow job is not able to start correctly (and the same error...

Error when installing apache beam in datalab instance in GCP using python2 kernel

I am trying to install apache beam in the datalab instance on GCP, but running into the following issue, would really love a way to fix this, I am not that proficient with either pip or datalab at...

installing tensorflow_transform and apache_beam on Datalab

I'm going over these example from google-cloud Coursera courses, and although they worked till a few weeks ago, I can't install tf.transform or apache_beam on Datalab...

Cloud Datastore batch insert entities while recognising if the same key already exist?

I'm trying to create a Cloud Dataflow 2.x (aka Apache Beam) PTransform to filter out elements of a PCollection that were already "seen" previously. The basic idea is that the PTransform gets a...

Issue testing GCP datastore

I have another spring boot project that is able to populate datastore in a test case but I am just running into runtime issues with this project and I cannot see what I am doing differently :( OS:...

Error importing cloud-spanner on apache beam 2.9

I am trying to update a few python apache beam scripts to the recent 2.9 version of apache beam. I am running into the following error when running a script that imports from google.cloud.spanner...

beam.io.ReadFromPubSub - ImportError: No module named iam.v1

I have a simple beam pipeline in which I am reading data from pub sub and writing to a file. I am running it on direct runner, the code is as follows: pubsub_data = ( p ...

Error creating universal sentence encoder embeddings using beam & tf transform

I have a simple beam pipline that takes some text and gets embeddings using universal sentence encoder with tf transform. Very similar to the demo made using tf 1. import tensorflow as tf import...

Error in pyarrow while installing Apache Beam in Raspberry Pi

I'm using a Raspberry Pi 3 B+ and should be installing the Apache Beam SDK to connect it to Google Cloud Platform services such as Pub/Sub, Dataflow, and BigQuery. I've got Raspbian GNU/Linux 10...