Set cache-control for entire S3 bucket automatically (using bucket policies?)

I need to set cache-control headers for an entire s3 bucket, both existing and future files and was hoping to do it in a bucket policy. I know I can edit the existing ones and I know how to...

Linux mount fails with error Transport endpoint not connected

From time to time for reasons unknown, the Amazon S3 Fuse mount on a linux server fails throughout the day. The only resolution is to umount and then mount the directory again. I tried writing...

Sonatype Nexus: How to use Amazon S3 as a storage for maven artifacts?

I've got a task to examine how to make our internal Nexus installation (installed on CentOs/RHEL) to store artifacts in Amazon S3 cloud storage (or any other cheap cloud storage). So far, I had...

Allowing permission using S3FS bucket directory for other users

I'm having problem using S3FS. I'm using [email protected]:~$ /usr/bin/s3fs --version Amazon Simple Storage Service File System 1.71 And I have the password file installed in the...

s3fs and Python os.walk

I'm trying to figure out a way to read images from an S3 bucket. Right now, my setup is to mount the bucket using s3fs, and then use a python script with os.walk to go through each individual...

MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

I am having this weird issue while working with AWS S3. I am working on application by which I can store the images to AWS bucket. Using Multer as middleware and S3FS library to connect and upload...

TypeError: __init__() got an unexpected keyword argument 'encoding'

Attempting a scrape of table data using pandas in Python 3.6 using Spyder3 on a MacBook Pro OS v10.13.2 (17C88). The code is: import pandas as pd ... url =...

s3fs gzip compression on pandas dataframe

I'm trying to write a dataframe as a CSV file on S3 by using the s3fs library and pandas. Despite the documentation, I'm afraid the gzip compression parameter it's not working with s3fs. def...

Python: recursive glob in s3

I am trying to get a list of parquet files paths from s3 that are inside of subdirectories and subdirectories of subdirectories (and so on and so forth). If it was my local file system I would do...

Mount S3 bucket as filesystem on AWS ECS container

I am trying to mount S3 as a volume on AWS ECS docker container using rexray/s3fs driver. I am able to do this on my local machine, where I installed plugin $docker plugin install...

Problems mounting a S3 bucket with s3fs

I am trying to mount a S3 bucket on an AWS EC2 instance following this instruction. I was able to install the dependencies via yum, followed by cloning the git repository, and then making and...

Automatically mounting S3 using s3fs on ubuntu 16

I am having an issue getting my s3 to automatically mount properly after restart. I am running an AWS ECS c5d using ubuntu 16.04. I able able to use s3fs to connect to my S3 drive manually...

Datatypes issue when convert parquet data to pandas dataframe

I have a problem with filetypes when converting a parquet file to a dataframe. I do bucket = 's3://some_bucket/test/usages' import pyarrow.parquet as pq import s3fs s3 =...

Writing pandas dataframe to S3 bucket (AWS)

I have an AWS Lambda function which queries API and creates a dataframe, I want to write this file to an S3 bucket, I am using: import pandas as pd import...

Trouble opening audio files stored on S3 in SageMaker

I stored like 300 GB of audio data (mp3/wav mostly) on Amazon S3 and am trying to access it in a SageMaker notebook instance to do some data transformations. I'm trying to use either torchaudio or...

After mounting S3 bucket system date of the directory is shown as 1970

I'm mounting S3 bucket using s3fs command, after mount the directory shows the system date as 1970. My google search could not lead to a fix, looking for a help. S3 Command: s3fs rsqatestbucket2...

Mounting an S3 bucket to an EC2 Ubuntu instance

I've looked at several tutorials on mounting S3 buckets using S3fs, but I sense these are geared towards key-based authentication. Right now if I "aws s3 ls" (or use other "aws s3 ls bucketname")...

aws glue `ImportError: cannot import name 'S3ArnParamHandler'`

I developed a pandas etl script locally and works fine. I prepared a wheel file and uploaded to s3. All packages are installed properly. However, when the script run, it shows ImportError: cannot...

Rex-ray AWS S3 external volume docker (docker-compose not working)

I am trying to use postgres and pgadmin with rex-ray external volume on AWS S3. I did: Docker plugin install rexray/s3fs:0.11.4 S3FS_ACCESSKEY=XXXXXXXXXXXXX S3FS_SECRETKEY=XXXXXXXXXXXXXXXXX And...

pytest How to mock s3fs.S3FileSystem open file

I am trying to mockup the call to open a file in a S3 bucket. The code that I have is: # mymodule.py import s3fs #... def __init__(self): self.s3_filesystem = s3fs.S3FileSystem(anon=False,...

How to stream a large gzipped .tsv file from s3, process it, and write back to a new file on s3?

I have a large file s3://my-bucket/in.tsv.gz that I would like to load and process, write back its processed version to an s3 output file s3://my-bucket/out.tsv.gz. How do I streamline the...

Trouble installing turbodbc

I am attempting to install turbodbc on my Ubuntu 20.10 machine. My specs are as follows: pip 20.2.4, Python 3.8.5 , gcc (Ubuntu 10.2.0-13ubuntu1) 10.2.0 I have attempted the solutions provided in...

Load XGBoost from an s3 bucket

I have an XGBoost model sitting in an AWS s3 bucket which I want to load. currently, I'm attempting to use s3fs to load the data, but I keep getting type errors: from s3fs.core import...

AWS SageMaker Processing job

I was able to run a simple python code in Notebook instance to read and write csv files from/to S3 bucket. Now I want to create the SageMaker processing job to run the same code without any...

Load CSV file into Pandas from s3 using chunksize

I'm trying to read a very big file from s3 using... import pandas as pd import s3fs df = pd.read_csv('s3://bucket-name/filename', chunksize=100000) But even after giving the chunk size it is...

ERROR: Could not build wheels for pandas which use PEP 517 and cannot be installed directly

I am using Docker with my dockerfile as: FROM python:3-alpine WORKDIR /app COPY ./requirements.txt . RUN apk update && apk add postgresql-dev gcc python3-dev musl-dev libffi-dev RUN pip install...

Access Smartsheet by column Name instead of Column Id

I am a newbie working with Smartsheet, I am trying to access the values of columns and store them in a list, and I am able to access the values by the index. In my use case, people can delete...

s3fs suddenly stopped working in Google Colab with error "AttributeError: module 'aiobotocore' has no attribute 'AioSession'"

Yesterday the following cell sequence in Google Colab would work. (I am using colab-env to import environment variables from Google Drive.) This morning, when I run the same code, I get the...

ERROR: Could not find a version that satisfies the requirement vineyard (from versions: none)

I am trying to install the package "grammar" whose dependencies include the packages "vineyard" and "Graphviz". I am using Pycharm, and I was able to install Graphviz without any issues. However,...

attributeerror: 'AioClientCreator' object has no attribute '_register_lazy_block_unknown_fips_pseudo_regions'

Recently, I have started to occupy the AWS platform, but when trying to occupy Sagemaker, the following error and I don't know if it is because of Sagemaker or it has something to do with...