How does one append large amounts of data to a Pandas HDFStore and get a natural unique index?

I'm importing large amounts of http logs (80GB+) into a Pandas HDFStore for statistical processing. Even within a single import file I need to batch the content as I load it. My tactic thus far...

ptrepack sortby needs 'full' index

I am trying to ptrepack a HDF file that was created with pandas HDFStore pytables interface. The main index of the dataframe was time but I made some more columns data_columns so that I can filter...

How to decrease size overhead of HDFStore?

I am experimenting with different pandas-friendly storage schemes for tick data. The fastest (in terms of reading and writing) so far has been using an HDFStore with blosc compression and the...

When reading huge HDF5 file with "pandas.read_hdf() ", why do I still get MemoryError even though I read in chunks by specifying chunksize?

Problem description: I use python pandas to read a few large CSV file and store it in HDF5 file, the resulting HDF5 file is about 10GB. The problem happens when reading it back. Even though I...

Cannot pip install the python module tables

I am trying to install tables so an existing python script does not complain when it tries to 'import tables' pip install tables Here is the output: Collecting tables Using cached...

When to use DataFrame.eval() versus pandas.eval() or Python eval()

I have a few dozen conditions (e.g., foo > bar) that I need to evaluate on ~1 MM rows of a DataFrame, and the most concise way of writing this is to store these conditions as a list of strings and...

TypeError: __init__() got an unexpected keyword argument 'encoding'

Attempting a scrape of table data using pandas in Python 3.6 using Spyder3 on a MacBook Pro OS v10.13.2 (17C88). The code is: import pandas as pd ... url =...

Reproducing conda environment when packages are no longer available from channels

I would like to publish the conda environment used for data analysis underlying a scientific paper. I saved the environment to a .yml file using conda env export > environment.yml I was able to...

Conda takes 20+ minutes to solve environment when package is already installed

NOTE: I'm duplicating this post because the question has been up on the conda github page for ~6-days with no response. The original link is...

zipline installation error : failed building wheel for bcolz

I'm trying to install zipline on a virtual environment on mac os. Python version = 3.6 / numpy, cython pre-installed When I try pip install zipline on the virtual environment, I get the following...

Python 3.7 anaconda environment - import _ssl DLL load fail error

I created anaconda environment with Python=3.7 and have trouble with the error of _ssl and DLL. When I tried to get back to my base environment, I have trouble getting the background processes to...

WEBP support not installed error with Pillow included in Anaconda

I have written a small code to open webp image in the Anaconda prompt. from PIL import Image im = Image.open('test.webp') It causes the following...

"AssertionError: Torch not compiled with CUDA enabled" in spite upgrading to CUDA version

I figured out this is a popular question, but still I couldn't find a solution for that. I'm trying to run a simple repo Here which uses PyTorch. Although I just upgraded my Pytorch to the latest...

No module named 'matplotlib.artist'

I faced with this error No module named 'matplotlib.artist' here is the complete error: --------------------------------------------------------------------------- ModuleNotFoundError ...

What does the as_json parameter for show_versions do in pandas?

When displaying version of Pandas library: print(pd.show_versions(as_json=True) OUTPUT: {'system': {'commit': None, 'python': '3.7.3.final.0', 'python-bits': 64, 'OS': 'Windows',...

"Verifying transaction: failed" Error in updating anaconda form anaconda prompt

I had been updating anaconda form "anaconda Prompt", i got following error i could not resolve the error can anyone please help me to resolve it Downloading and Extracting Packages libxslt-1.1.33 ...

What does the PyTables warning "a closed node found in the registry" mean?

When using pandas.to_hdf function to save data to a HDF5 file, I'm getting the following warning: C:\{my path to conda environment}\lib\site-packages\tables\file.py:426: UserWarning: a closed...

Problem with creating an environment from .yml file, error "CondaEnvException: Pip failed" raised

I am trying to create an environment based on a .yml file, the name of the file is env.yml . I run the following snippet on terminal: conda env create -f env.yml Then anaconda starts...

virtualenv: pre-installed with packages on creating virtual environment

I created a virtual environment using command virtualenv env on my terminal. On using pip freeze > requirements.txt after activating the virtual environment, I was a bunch of packages...

Why does `conda list cudnn` have no output after `conda install pytorch torchvision cudatoolkit=10.2 -c pytorch` installation

*Please feel free to vote "Reopen" at the bottom of this question. The reason is that I have marked this as a duplicate although the answers there are not clear enough for this question.* As soon...

Conda: How can I update just the packages I specify in my command?

I'm trying to update a single package in my latest Anaconda installation (release date 2020-07-31, Ubuntu 20.04.1), but lots of other package updates were suggested, including a few I don't...

"ImportError: No module named seaborn" in Azure ML

Created a new compute instance in Azure ML and trained a model with out any issue. I wanted to draw a pairplot using seaborn but I keep getting the error "ImportError: No module named seaborn" I...

Ipywidgets with Voila not showing: ERROR tornado Uncaught exception GET

I am trying to use voila by running the examples they provide, but the widgets don't show (outputs from jupyter and voila) and I get these errors: (voila_env) Z:\Programming\voila\notebooks>voila...

Importing the numpy C-extensions failed for embedded Python code

I seem to have a problem importing numpy only when trying to run my C++ code with embedded Python. The following code is just a dummy code that captures the problem. Python: from __future__ import...

It seems that scikit-learn has not been built correctly

I have been using Jupyter Notebook for my machine learning project. Before scikit-learn was working fine but eventually I installed pip install imblearn and pip install -U imbalanced-learn after...

Installing python tables on mac with m1 chip

I am trying to use tables in python3 on a new mac mini with the M1 chip. I am getting multiple errors when running HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.12.0_1 pip3 install tables ERROR:...

Failed to run TA-lib Python on Heroku

I want to deploy an app using Heroku but it did not manage to build it. See my build logs under this message. It looks that my problem is that it cannot load/install Ta-Lib (package that I want to...

Jupyter Notebook Cannot Connect to Kernel, Likely due to Zipline / AssertionError

All of my virtual environments work fine, except for one in which the jupyter notebook won't connect for kernel. This environment has Zipline in it, so I expect there is some dependency that is a...

Updating packages in conda

I have a problem with updating packages in conda. The list of my installed packages is: # # Name Version Build Channel _anaconda_depends 2020.07 ...

How can I check if c-blosc is installed using the command line?

Using command -v c-blosc returns nothing, even though it's installed c-blosc describes themselves as a compression library, so it's not a command A few things I've tried % c-blosc zsh: command...