Replace pickle in Python multiprocessing lib

I need to execute the code below (simplified version of my real code base in Python 3.5): import multiprocessing def forever(do_something=None): while True: do_something() p =...

one hot encoding for frequent values only

I am looking to do one hot encoding to a column, but only for those that are very frequent. All that are below a threshold T will be put in their own category. My strategy was to create a...

Pickle file works on one machine and doesn't work when opened on other

I work remotely on a cluster to which I connect via ssh. I run some tasks there and then I use pickle to save objects like that: with open('tst.pkl','wb') as g: pickle.dump(pkl_pack, g) Then...

PyTorch Cuda with anaconda not available

I'm using anaconda to regulate my environment, for a project i have to use my GPU for network training. I use pytorch for my project and i'm trying to get CUDA working. I installed cudatoolkit,...

Unable to install/run docker with opencv

I'm using the code below in Dockerfile and it builds successfully but it does not run. How can I get it to work? FROM python:3.5-slim COPY . /app WORKDIR /app RUN apt-get update RUN apt-get -y...

Parallel: Import a python file from sibling folder

I have a directory tree working_dir\ main.py my_agent\ my_worker.py my_utility\ my_utils.py Code in each file is as follows """ main.py """ import os,...

How to assuredly suppress a DeprecationWarning in Python?

I believe this question was raised lots of times already but I have a specific use case where I can't solve the issue with many of the methods described on the web. In one of my projects, I am...

Python: why do I get an error when I import geopandas?

I try to import geopandas in my jupyter notebook. import geopandas as gp and I get the following error: ImportError:...

Cuda Runtime Error cudaErrorNoDevice: no CUDA-capable device is detected

I am using Chainer, Cupy for CUDA 8.0 . I am trying to train machine learning model using python3.5 script, but I got this error: cupy.cuda.runtime.CUDARuntimeError: cudaErrorNoDevice: no...

WEBP support not installed error with Pillow included in Anaconda

I have written a small code to open webp image in the Anaconda prompt. from PIL import Image im = Image.open('test.webp') It causes the following...

spacy with joblib library generates _pickle.PicklingError: Could not pickle the task to send it to the workers

I have a large list of sentences (~7 millions), and I want to extract the nouns from them. I used joblib library to parallelize the extracting process, like in the following: import spacy from...

"AssertionError: Torch not compiled with CUDA enabled" in spite upgrading to CUDA version

I figured out this is a popular question, but still I couldn't find a solution for that. I'm trying to run a simple repo Here which uses PyTorch. Although I just upgraded my Pytorch to the latest...

HDF5 library version error - HDF5 ver 1.10.4

I'm trying to import some packages with spyder (OS x64), Anaconda and pyton 3.x The error is pretty famous in the internet. The solution proposed is to match the version of the library 1.10.5 with...

UnsatisfiableError: The following specifications were found to be incompatible with each other:

I am running on Ubuntu 16.04 and I have run into issues with Anaconda. Due to package inconsistencies, I can no longer install, update or remove packages. $ conda info NVIDIA: no NVIDIA devices...

Create conda environment: "Found conflicts!" when solving environment and "Finding shortest conflict path" running forever

I was provided with an environment.ubuntu.yml file to be able to create a conda environment. However, running conda create env --file environment.ubuntu.yml I get the following output: conda env...

ERROR: You must give at least one requirement to install -- when running: pip install --upgrade --no-binary hdbscan

I am trying to install hdbscan in my PC which runs Windows 10 and has installed Python 3.6. My first attempt failed: (base) C:\WINDOWS\system32>pip install hdbscan --user Collecting hdbscan ...

When I open the spyder in anaconda, the spyder indicates that an error ocurred while starting the kernel in spyder

When I open the Spyder in anaconda, the Spyder indicates that an error occurred while starting the kernel in Spyder. I have tried (in the terminal) two suggested commands from Spyder. However,...

PySpark and Protobuf Deserialization UDF Problem

I'm getting this error Can't pickle <class 'google.protobuf.pyext._message.CMessage'>: it's not found as google.protobuf.pyext._message.CMessage when I try to create a UDF in PySpark....

AttributeError: module 'tensorflow' has no attribute 'compat' when loading tf.compat.v1.train.SessionRunHook

I can see that this question has been asked before here tensorflow-has-no-attribute-compat but the answer given was to Microsoft Visual C++ 2015-2019 Redistributable (x64) It did not work for the...

Stable Baselines doesn't work with tensorflow

So I recently got back into machine learning, and decided to start the Kaggle course for "ConnectX" (https://www.kaggle.com/learn/intro-to-game-ai-and-reinforcement-learning). I am trying to do...

virtualenv: pre-installed with packages on creating virtual environment

I created a virtual environment using command virtualenv env on my terminal. On using pip freeze > requirements.txt after activating the virtual environment, I was a bunch of packages...

pipx-installed Spyder doesn't recognize Poetry-installed pandas only when trying to view DataFrame in Variable Explorer

I'm using Python 3.8.5. My OS is Kubuntu 18.04. I installed Spyder and Poetry via pipx: pipx install spyder --include-deps pipx install poetry Within my project folder, I installed pandas and...

Trouble installing turbodbc

I am attempting to install turbodbc on my Ubuntu 20.10 machine. My specs are as follows: pip 20.2.4, Python 3.8.5 , gcc (Ubuntu 10.2.0-13ubuntu1) 10.2.0 I have attempted the solutions provided in...

Using apache-airflow-providers-snowflake on airflow (no module named Snowflake)

I have installed package apache-airflow-providers-snowflake on airflow on docker and i am getting error No module named Snowflake Please refer attachment (check the error mentioned for the...

Conda - how to update only cudatoolkit in an existing environment?

This is a specific instance of a general problem that I run into when updating packages using conda. I have an environment that is working great on machine A. I want to transfer it to machine...

This version of ChromeDriver only supports Chrome version 87 Current browser version is 90.0.4430.93

ChromeDriver is giving an error about its version being too low: This version of ChromeDriver only supports Chrome version 87 Current browser version is 90.0.4430.93. I've updated pip's packages...

Raytune is throwing error: "module 'pickle' has no attribute 'PickleBuffer'" when attempting hyperparameter search

I am more or less following this example to integrate the ray tune hyperparameter library with the huggingface transformers library using my own dataset. Here is my script: import ray from ray...

Python AWS Lambda Error "libgomp.so.1: cannot open shared object file: No such file or directory" from AWS EFS file system mounted on Ubuntu

I have an AWS lambda function connected to an AWS EFS mounted on a EC2 instance of Ubunutu 18. I am getting this error below when importing a LightGBM model i believe. { "errorMessage":...

How to install PyCaret in AWS Glue

How can I properly install PyCaret in AWS Glue? Methods I tried: --additional-python-modules and --python-modules-installer-option Python library path easy_install as described in...

Python: matplotlib.pyplot.show() is not showing the plot

import matplotlib.pyplot as plt plt.plot([1,2,3]) plt.show() input("Press enter to continue...") If I run the script above within VS Code, or even within command prompt, the plot canvas only...