Why does my Python program average only 33% CPU per process? How can I make Python use all available CPU?

I use Python 2.5.4. My computer: CPU AMD Phenom X3 720BE, Mainboard 780G, 4GB RAM, Windows 7 32 bit. I use Python threading but can not make every python.exe process consume 100% CPU. Why are they...

Starting a separate process

I want a script to start a new process, such that the new process continues running after the initial script exits. I expected that I could use multiprocessing.Process to start a new process, and...

Mulitprocess Pools with different functions

Most examples of the Multiprocess Worker Pools execute a single function in different processes, f.e. def foo(args): pass if __name__ == '__main__': pool =...

Python decorator with multiprocessing fails

I would like to use a decorator on a function that I will subsequently pass to a multiprocessing pool. However, the code fails with "PicklingError: Can't pickle <type 'function'>: attribute lookup...

How to use multiprocessing queue in Python?

I'm having much trouble trying to understand just how the multiprocessing queue works on python and how to implement it. Lets say I have two python modules that access data from a shared file,...

Memory usage keep growing with Python's multiprocessing.pool

Here's the program: #!/usr/bin/python import multiprocessing def dummy_func(r): pass def worker(): pass if __name__ == '__main__': pool = multiprocessing.Pool(processes=16) for...

Differences in projecting satellite images with cartopy and pyresample

I have written a python script to project and overlay geostationary satellite images from the university of Dundee so the resulting image can be used for xplanet to render the surface of the...

How to speed up process of loading and reading JSON files in Python?

I am running a script (in multiprocessing mode) that extract some parameters from a bunch of JSON files but currently it is very slow. Here is the script: from __future__ import print_function,...

Making SVM run faster in python

Using the code below for svm in python: from sklearn import datasets from sklearn.multiclass import OneVsRestClassifier from sklearn.svm import SVC iris = datasets.load_iris() X, y = iris.data,...

Is it possible to use multiprocessing in a module with windows?

I'm currently going through some pre-existing code with the goal of speeding it up. There's a few places that are extremely good candidates for parallelization. Since Python has the GIL, I thought...

Memory usage steadily growing for multiprocessing.Pool.imap_unordered

I just noticed that my program is using more and more memory as it processes a large file. It's only processing one line at a time, though, so I couldn't figure out why it would keep using more...

Python multiprocessing queue get() timeout despite full queue

I am using Python's multiprocessing module to do scientific parallel processing. In my code I use several working processes which does the heavy lifting and a writer process which persists the...

How to store return value of function in multiprocessing process in python?

I am writing a python program which executes functions in parallel manner. Here is code: from multiprocessing import Process def sqr(args): results = [] for i in args: ...

Stopping a Python Multiprocessing BaseManager 'serve_forever' server?

I have the following setup in Python 3.6 for interprocess communication: from multiprocessing.managers import BaseManager class MyManager(BaseManager): pass MyManager.register('get_instance',...

Python wait for processes in multiprocessing Pool to complete without either closing Pool or use map()

I have a code piece like below pool = multiprocessing.Pool(10) for i in range(300): for m in range(500): data = do_some_calculation(resource) pool.apply_async(paralized_func,...

parallelize genetic algorithm with numba (cuda)

I am making a genetic algorithm on neural nets playing snake for educational purposes I want to be able to run my python code on my gpu (cuda enabled) in order to save some time and accelerate the...

Why is the curve of my permutation test analysis not smooth?

I am using a permutation test (pulling random sub-samples) to test the difference between 2 experiments. Each experiment was carried out 100 times (=100 replicas of each). Each replica consists of...

How to synchronize Python processes on milliseconds level?

I'm trying to run two Python functions on two cores exactly at the same time. Each process runs a very long loop (theoretically infinite loop). It is important they stay synchronized at the same...

python3 queue.put() blocking main

python3 queue.put() will block main process when queue size over particular value(1386). I use 30 subprocesses and two queue to count an int number, each subprocess get number from first queue and...

spacy with joblib library generates _pickle.PicklingError: Could not pickle the task to send it to the workers

I have a large list of sentences (~7 millions), and I want to extract the nouns from them. I used joblib library to parallelize the extracting process, like in the following: import spacy from...

Running a ProcessPoolExecutor in IPython

I was running a simple multiprocessing example in my IPython interpreter (IPython 7.9.0, Python 3.8.0) on my MacBook and ran into a strange error. Here's what I typed: [In [1]: from...

Python 3 asyncio with aioboto3 seems sequential

I am porting a simple python 3 script to AWS Lambda. The script is simple: it gathers information from a dozen of S3 objects and returns the results. The script used multiprocessing.Pool to gather...

Multiprocessing memory leak, processes that stay around forever

I'm trying to solve a multiprocessing memory leak and am trying to fully understand where the problem is. My architecture is looking for the following: A main process that delegates tasks to a...

Why does joblib parallel execution make runtime much slower?

I want to shuffle values in a 3D numpy-array, but only when they are > 0. When I run my function with a single core, it is much faster than with even 2 cores. It is way beyond the overhead of...

Inter-Process Communication in dotnet 5/6

I am developing a new project in Blazor WebAssembly with .NET 5, this ASP.NET Core hosted service (server part of Blazor app) will need to talk to a .NET Framework 4.7.2 Windows Service. The...

ProcessPoolExecutor within class object Python.causes : cannot pickle 'weakref' object

I have the following piece of code, which throws 'cannot pickle 'weakref' object'. I've seen tons of examples, where people wrap the ProcessPoolExecutor with a if __name__ == '__main__', however...

gsutil timeout in every call refreshing access_token

This might be a duplicate but none of the previous answers match my conditions. I installed gsutil as part of the google-cloud-sdk following https://cloud.google.com/sdk/docs/install. I could...

"Can't parse 'pt'. Sequence item with index 0 has a wrong type." PointPolygon error

I'm getting this error when I tried to use OpenCV for a project. File "/Users/alex/opt/anaconda3/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args,...

Ansible cannot pickle 'module' object

Running the playbook with $ ansible-playbook test-playbook.yml Getting error response ERROR! Unexpected Exception, this is probably a bug: cannot pickle 'module' object Also have tried shell:...

Run portion of Python code in parallel from a Streamlit app

I have a Streamlit app that configures a batch of energy models to run and a separate module of code that builds and runs those models in parallel. The exiting code currently uses SCOOP...