Calculating CRC16 in Python

I'm trying to evaluate appropriate checksum based on CRC-16 algorithm using crcmod Python module and 2.7 version of Python interpreter. The checksum parameters are: CRC order: 16 CRC polynomial:...

Recreating a C CRC32 algorithm in Python with crcmod

I'm attempting to decode a Novatel GPS stream in Python, and I have all of the components for the sync, header, and payload sorted, but I cannot replicate the CRC32 used as part of the messages....

Difficulty comparing generated and google cloud storage provided CRC32c checksums

I am attemptting to get a CRC32c checksum on my local file so I can compare it to the blob.crc32c provided by the gcloud library. Google says I should be using the crcmod module in order to...

How to install Google Cloud SDK's app-engine-python on Compute Engine (Ubuntu 16.04)?

I would like to use Google Cloud SDK (including the Python Extensions for App Engine) inside Compute Engine, so I can replicate my local development setup in a virtual machine. I.e. run local...

Google cloud: Using gsutil to download data from AWS S3 to GCS

One of our collaborators has made some data available on AWS and I was trying to get it into our google cloud bucket using gsutil (only some of the files are of use to us, so I don't want to use...

What should I do about this gsutil "parallel composite upload" warning?

I am running a python script and using the os library to execute a gsutil command, which is typically executed in the command prompt on Windows. I have some file on my local computer and I want to...

Error while using tensorflow transform' writeTransform function

I'm currently using Tensorflow transform library to convert and save the transformation, though it used to work before just fine currently im facing a bit of issue something similar to below I...

Installing AIORTC via pip on Windows10

how can I install AIORTC, the WebRTC/ObjectRTC bindings for Python on windows? when I do pip install aiortc, I get the below error: I have the lib opus source code I got from their webpage, but...

Python CRC8 calculation

I have some bytes I want to calculate the CRC8 on in python. I don't have such an experience with that but I know, from the tech specs of my device, that this calculation has to be made with a...

Installing google cloud text-to-speech on rasberry pi 3 b+ issues dealing with hashlib

I'm trying to install the google cloud sdk/google cloud text to speech on my Rasberry Pi 3+ and have tried do so for about 3 days now. I installed python 2.7.9 only to find out it was already...

How to remove a multi line block of text from $pattern in Powershell

I'm getting the contents of a text file which is partly created by gsutil and I'm trying to put its contents in $body but I want to omit a block of text that contains special characters. The...

gsutil rsync from google cloud storage to s3 hangs on large file

I am experiencing a hang on a larger file (560MB) when attempting to backup my google cloud storage bucket to aws s3 bucket using the following command: gsutil -m rsync -rd gs://<MyGoogleBucket>/...

beam.io.ReadFromPubSub - ImportError: No module named iam.v1

I have a simple beam pipeline in which I am reading data from pub sub and writing to a file. I am running it on direct runner, the code is as follows: pubsub_data = ( p ...

CRC8 Checksum with custom polynom in Python crcmod

I'm trying to use crcmod to compute a crc checksum sent by a microcontroller with a custom polynom of 183 (decimal). When I try to use crc8 = crcmod.Crc(183) I get ValueError: The degree of the...

why (ModuleNotFoundError: No module named 'scapy') shows up?

I wrote a program using python3 which read and responses data via DNP3 protocol, It is intended to be run on a rpi3 I wrote the code on my laptop then took it to the rpi, installed all the...

Import Error: No module named 'google.cloud' on ApacheBeam

I get an import error with importing apache beam's google datastore api.I have one version of Python 3 installed on my Windows 10 64-bit system. Can somebody help me? I have try to solve it but i...

How can I use CRC/CRC libs in python?

I'm trying to use the checksum algorithm CRC in my code (python), but I'm facing several dificulties. I tried to implement a CRC algorithm by my own but it's completely bad, a file of 20MB cost me...

Gsutil Always Time Out

I am just learning the Google Cloud with Keras, but once when I need to use Gsutil to submit a project, it always is time out. I feel confused about. It seems not a networking problem, I can...

Error creating universal sentence encoder embeddings using beam & tf transform

I have a simple beam pipline that takes some text and gets embeddings using universal sentence encoder with tf transform. Very similar to the demo made using tf 1. import tensorflow as tf import...

ImportError: cannot import name 'downsample' while importing lasagne in python 3.6

I am getting the above error with the following import statements on Google Colab GPU: import argparse #import cPickle import _pickle as cPickle import time import os import numpy as np import...

Error in pyarrow while installing Apache Beam in Raspberry Pi

I'm using a Raspberry Pi 3 B+ and should be installing the Apache Beam SDK to connect it to Google Cloud Platform services such as Pub/Sub, Dataflow, and BigQuery. I've got Raspbian GNU/Linux 10...

Failed to build pyarrow (for python 3.7) with an error message ERROR: Could not build wheels for pyarrow which use PEP 517

I am trying to install apache-beam[gcp] in my virtual env. My python version is 3.7 My pip3 version is 20.0.2 when do try to install ( without sudo) and run the following:- pip3...

Gsutil copy to local is failing due to composite object requires integrity checking with CRC32c,

I am facing a problem while copying the file from google cloud to local directory. File size is of 200MB. I was able to copy the file without any issues till few weeks earlier but suddenly seeing...

Google cloud gsutil fails inconsistently due to crcmode not compiled

I'm running my build with google cloudbuild and I have a step that copies cached files from a google storage bucket. the step is using google's gsutil image from...

How to deploy Google Cloud Dataflow with connection to PostgreSQL (beam-nuggets) from Google Cloud Functions

I'm trying to create ETL in GCP which will read part of data from PostgreSQL and put it in the suitable form to BigQuery. I was able to perform this task deploying Dataflow from my computer, but I...

How to upload large docker images to google container registry?

I'm creating a lap image from this (https://github.com/ulsmith/alpine-apache-php7) and in my application, I have a 2.5 GB folder of images the image is built successfully but when I try to push it...

Dataflow Flex template official tutorial is not working

I am trying to reproduce this tutorial to run a Flex Template on Dataflow. Few minutes after submitting the job, it fails with this error: Output from execution of subprocess: b'Collecting...

gsutil timeout in every call refreshing access_token

This might be a duplicate but none of the previous answers match my conditions. I installed gsutil as part of the google-cloud-sdk following https://cloud.google.com/sdk/docs/install. I could...

Sudden Tensorflow / Keras Google Colab dependency problems `AttributeError: module 'tensorflow._api.v1.compat.v2' has no attribute '__internal__'`

I have running a machine learning model (Matterport's Mask R-CNN) in google colab for a couple of weeks. All of a sudden today I am unable to run any of my notebooks due to I think some kind of...