Faster s3 bucket duplication

I have been trying to find a better command line tool for duplicating buckets than s3cmd. s3cmd can duplicate buckets without having to download and upload each file. The command I normally run...

What is the algorithm to compute the Amazon-S3 Etag for a file larger than 5GB?

Files uploaded to Amazon S3 that are smaller than 5GB have an ETag that is simply the MD5 hash of the file, which makes it easy to check if your local files are the same as what you put on S3. But...

Compress file on S3

I have a 17.7GB file on S3. It was generated as the output of a Hive query, and it isn't compressed. I know that by compressing it, it'll be about 2.2GB (gzip). How can I download this file...

How can I download a file from an S3 bucket with wget?

I can push some content to an S3 bucket with my credentials through S3cmd tool with s3cmd put contentfile S3://test_bucket/test_file Question is, I am required to download the content from this...

filtered results with s3cmd get

I'm using cmd tool for AWS S3 called s3cmd. I'm trying 'get' set of folders filtered by front part of name directory name like '/192.168.*/'. Basically I have S3 bucket with a lot of directories...

Fast way to get AWS S3 key count in bucket

Is there anyone out there knowing a fast way to get the count of my keys in S3? I usually do s3cmd ls s3://bucket/ | wc -l but my bucket contains a huge number of keys which makes this operation...

how to add cache control in AWS S3?

I have moved 20000 files to AWS S3 by s3cmd command. Now i want to add cache-control for all images (.jpg) These files are located in ( s3://bucket-name/images/ ). How can i add cache-control for...

s3cmd delete after upload

I am trying to upload files from a folder on my machine onto s3 and delete the file from my local machine after the upload is complete. I have a cron running every minute with this command, but it...

s3cmd to set correct content type for image/jpeg

I have uploaded a number of directories that contain images, what is the correct way to modify the content-type to "image/jpeg" without having to re-upload the images? I tried: s3cmd modify...

How to download data from Amazon's requester pay buckets?

I have been struggling for about a week to download arXiv articles as mentioned here: http://arxiv.org/help/bulk_data_s3#src. I have tried lots of things: s3Browser, s3cmd. I am able to login to...

PgSQL - How to import database dump only when database completely empty?

The use-case actually to automate this with ansible. I want to import database dump only when database is completely empty (no tables inside). Of course there's always a way to execute sql...

How do I delete a versioned bucket in AWS S3 using the CLI?

I have tried both s3cmd: $ s3cmd -r -f -v del s3://my-versioned-bucket/ And the AWS CLI: $ aws s3 rm s3://my-versioned-bucket/ --recursive But both of these commands simply add DELETE markers...

Download file from S3 to Rails 4 app

I have an AWS VM that runs a daily task and generates several files. I want my Rails app to download these files and place them in a folder within the app. Is there a gem or method in Ruby that...

Why does file uploaded to S3 have content type application/octet-stream unless I name the file .html?

Even though I set content type to text/html it ends up as application/octet-stream on S3. ByteArrayInputStream contentsAsStream = new ByteArrayInputStream(contentAsBytes); ObjectMetadata md = new...

Running AWS CLI through Python returns a "sh: 1: aws: not found" error

I am trying to copy a file into an S3 bucket, using Python, like so: cmd = 'aws s3 %s %s' % (filename, bucketname) os.system(cmd) It gives me a sh: 1: aws: not found error. However, using s3cmd...

Exclude multiple folders using AWS S3 sync

How to exclude multiple folders while using aws s3 syn ? I tried : # aws s3 sync s3://inksedge-app-file-storage-bucket-prod-env \ ...

How do you invalidate cache of index.html for a static site hosted on S3 with cloudfront?

So I have hosted my angular app on s3 with a cloudfront dist. I do file revision-ing (using grunt filerev) to make sure that I never get stale content. But, how should I version the index.html...

How to change storage class of object in s3 bucket?

I need to change storage class of a s3 object from STANDARD class into STANDARD_IA using s3cmd. Can anybody help me to do this task. Thanks in advance.

S3 Sync Not Writable Warning

I am trying to use s3cmd sync (vanilla command remote to local), but for each file I get a not writable: Operation not permitted warning (see below). The files appear to download correctly,...

Configuring s3cmd non-interactively through bash script

I am trying to configure s3cmd. When at the command line, s3cmd --configure runs the wizard and the data entry is straight forward. Now, I want to run this through a bash script. I used: s3cmd...

AWS S3 permissions - error with put-bucket-acl

I am trying to move an S3 bucket from one account (A) to another (B). I have succeeded with that operation and remove the bucket from account A. I am trying to move the new bucket from account B...

AWS S3: How to check if a file exists in a bucket using bash

I'd like to know if it's possible to check if there are certain files in a certain bucket. This is what I've...

How to Generate a Presigned S3 URL via AWS CLI

Is there a way to create presigned URL for objects in S3 bucket using AWS CLI? I know that could be done using SDK, but is it possible with CLI? I found this on one of the AWS docs, but can't...

Digital Ocean Spaces | Add expire date for files with s3cmd

I try add expire days to a file and bucket but I have this problem: sudo s3cmd expire s3://<my-bucket>/ --expiry-days=3 expiry-prefix=backup ERROR: Error parsing xml: syntax error: line 1,...

AWS Bucket Policy Error: Policy has invalid action

I have a very basic goal: to share all content of my bucket to a list of specific users, read only. This used to work with a tool called s3cmd. All I need to do was to add a user (identified by...

How to delete files older than 7 days in Amazon S3?

Need to delete files in Amazon S3 which is older than 7 days, need a shell script to do this, no luck with google search, i found the below...

Need to transfer data from AWS EFS to EFS in different regions

I have an EFS configured in the oregon region but as the EFS is now available in the Tokyo region so I want to transfer all EFS data to the Tokyo region where EFS is already created. AWS doc...

Why is Module python-magic not available?

I am using s3cmd to sync to an S3 bucket from a Mac and I am getting the following warning: WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. But pip...

how to download partial content from remote video url in nodejs?

I have the following code to download a specific part (range) of a video using nodejs http.get(), const fileUrl = 'https://www.example.com/path/to/video.mp4' const fs = require('fs') const http =...

Getting error while writing dataframe to Ceph Storage

In my organisation I'm currently exploring how we can use Ceph to replace HDFS to run out AI/ML workloads. As part of this initiative we setup a Ceph Cluster and imported it into Kubernetes using...