I have a very long and repetitive python path, where do I look to correct this?

I know it is probably not necessary to paste the whole path, but just for the record I have done so below. Whenever I run a python command, it takes a long time to load this path I suppose. I have...

Pympler summary doesn't seem to make sense

I'm doing some sanity checks with Pympler to make sure that I understand results when I try to profile an actual script, but I'm a bit puzzled at the results. Here are the sanity checks I've...

Memory leak with loop-based Seaborn plotting in Jupyter notebook

I am having some trouble managing memory with a function that is being called while in a Jupyter notebook. This function contains a calculation which does some pairwise data analysis. During the...

Import works in Python but not IPython

No problem when I import in a python terminal: $ python Python 3.5.4 | packaged by conda-forge | (default, Aug 10 2017, 01:41:15) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on...

Python: how to get size of all objects in current namespace?

I have some code that I am running from my own package and the program is using a lot more memory (60GB) than it should be. How can I print the size of all objects (in bytes) in the current...

How to create new environment from a text file without environment name?

I am trying to create a new environment in Anaconda from a given file provided by Udacity here. However, I keep getting this error: SpecNotFound: Can't process without a name I think this is due...

How to detect memory leak in python code?

I'm new to machine learning and python both! I want my code to predict the object which is mostly the car in my case. When I start the script it runs smoothly but after 20 or so pictures it hangs...

How to detect Python memory leaks

We are trying to tune some parameters in Python on Windows 10. We have many parameters and the range is wide so we are trying to calculate around 1000000 possibilities at least. We are using...

Why do namedtuples use less memory than dictionaries?

I'm asking this because I found it surprising -- I thought a namedtuple would have more overhead. (The background is I was caching a large Django query in memory and found Django objects to be...

Tensorflow 2.0 leaking memory when `input_shape` is specified in keras layer

I'm using Tensorflow 2.0 with keras APIs. There are situations where I need to generate multiple models within the same program execution (say, a cross-validation type of situation). When I...

delay caused by fat argument in python multiprocessing

i am using multiprocessing in python to parallel some computing-heavy functions. but i found that there is a delay in process creating if passing a fat argument (e.g., a 1000-note networkx graph...

UnsatisfiableError: The following specifications were found to be incompatible with each other:

I am running on Ubuntu 16.04 and I have run into issues with Anaconda. Due to package inconsistencies, I can no longer install, update or remove packages. $ conda info NVIDIA: no NVIDIA devices...

Python memory leak after del

So I know this has been addressed a number of times,...

Installed Spyder 4.0 using anaconda now it fails to open

I uninstalled my old version of spyder and installed spyder 4.0 using anaconda navigator. When I try to run spyder I get the following error message. What do I need to do to install spyder 4.0...

ContextualVersionConflict when importing guesslang package

When importing guesslang package, I've got ContextualVersionConflict error about html5lib package. ContextualVersionConflict: (html5lib 1.0.1...

How can I improve the performance of my script?

I have a "seed" GeoDataFrame (GDF)(RED) which contains a 0.5 arc minutes global grid ((1802)(360*2) = 259200). Each cell contains an absolute population estimate. In addition, I have a "leech" GDF...

Why does Python garbage collection with threads not work?

When I define a thread as an instance variable, other objects defined are not garbage collected. In this example an array buffer is created in each iteration of the loop that is never garbage...

TensorFlow -- Duplicate plugins for name projector -- Anaconda Prompt

I am trying to run a generic analysis of mnist, as shown below. from __future__ import absolute_import from __future__ import division from __future__ import print_function import math import...

Debug CPython memory fragmentation

I have a long-running CPython 3.8 process. After a while, it's using a huge amount of RAM. I have tried running gc.collect() using pympler to discover all known Python objects import gc import...

how do we resolve multiple package conflicts in coda

I am trying to install OpenCV in Anaconda. The installation fails due to a conflict of packages. Below is the list of conflicts: Package lz4 conflicts for: libarchive ->...

Memory-efficient data structure for a set of integer tuples

I have a computational problem that involves a huge amount of integer tuples which I have to store in a set. In more details, the setting is: There is a positive integer n (<=10) and a lot of...

memory_profiler while using lambda expression to connect slots

I am experimenting memory_profiler in Python3 following https://medium.com/zendesk-engineering/hunting-for-memory-leaks-in-python-applications-6824d0518774 thanks to @eyllanesc here PyQt5...

muppy: Getting the name of the biggest PyObjects in a Python program sorted by size

I have a memory leak issue with my program. I'm using muppy from the pympler library to print a list of the biggest PyObjects in my program sorted by size in bytes. Here is a reproducible example...

Python mmap "memory leak"

I have a reasonably large file (~4gb on disk) that I want to access with Python's mmap module to gain some familiarity with memory maps. I have a 64 bit system, and am running something similar to...

How to track a memory-usage growing object in a python script

I have a python script, whose memory usage (observed in top) is growing indefinitely when the script is running. My script is not supposed to store anything. It just receives some request (REST...

Memory leak issue using PyTorch IterableDataset with zarr

I'm trying to build a pytorch project on an IterableDataset with zarr as storage backend. class Data(IterableDataset): def __init__(self, path, start=None, end=None): super(Data,...

Multiprocessing memory leak, processes that stay around forever

I'm trying to solve a multiprocessing memory leak and am trying to fully understand where the problem is. My architecture is looking for the following: A main process that delegates tasks to a...

How do I install pip packages through a conda-build recipe?

I am trying to make a conda package out of a data pipeline. The idea is to distribute the package locally through our department for quick deployment. Our pipeline uses 2 packages that are not...

Queue.get() memory leak

I use the following code to produce what looks like a memory leak after emptying a queue with the get() method. import queue import os import psutil def run(del_after_puts, del_after_gets,...

Ubuntu 16.04 : Error "Could not connect to archive.ubuntu.com:80 (91.189.88.152)" while running "apt-get update " command in a dockerfile

I will put across my issue as follows : I want to build a docker image for Hyperledger Indy-sdk While building docker image it uses docker-compose command to build two images and combine them...