Getting S3 objects' last modified datetimes with boto

I'm writing a Python script that uploads files to S3 using Boto. I only want to upload changed files (which I can check by their "last modified" datetimes), but I can't find the Boto API endpoint...

How can tox install the modules via the requirements file?

Our python project has a requirements.txt file which lists some dependent module. We used to use pip install -r requirements.txt to install these dependencies. We are now using tox to build up...

How to launch EC2 instance with Boto, specifying size of EBS?

I'm using boto/python to launch a new EC2 instance that boots from an EBS volume. At the time I launch the instance, I'd like to override the default size of the booting EBS volume. I found no...

Python Django requirements.txt

I have a requirements.txt file containing all my dependencies but it is not processed correctly : After a pip install -r requirements.txt, I get the following pip freeze...

Python: PIP install path, what is the correct location for this and other addons?

I have recently had a few problems with my Python installation and as a result I have just reinstalled python and am trying to get all my addons working correctly as well. I’m going to look at...

Python boto ec2 - How do I wait till an image is created or failed

I am writing a code to iterate through all available instances and create an AMI for them as below: for reservation in reservations: ...... ami_id = ec2_conn.create_image(instance.id,...

Move files between two AWS S3 buckets using boto3

I have to move files between one bucket to another with Python Boto API. (I need it to "Cut" the file from the first Bucket and "Paste" it in the second one). What is the best way to do that? **...

How to create EC2 instance through boto python code

requests = [conn.request_spot_instances(price=0.0034, image_id='ami-6989a659', count=1,type='one-time', instance_type='m1.micro')] I used the following code. But it is not working.

PyCharm intellisense for boto3

having problems seeing full intellisense (code completion) options in PyCharm. working with python 3.4 on Windows. the suggests are partially working: import boto3 s = boto3.Session() (boto3....

What is the difference between the AWS boto and boto3

I'm new to AWS using Python and I'm trying to learn the boto API however I noticed that there are two major versions/packages for Python. That would be boto and boto3. What is the difference...

How to configure Elasticsearch to pull messages from Amazon SQS

In my current scenario , i am having ElasticSearch on a ubuntu 14.04 machine and a amazon SQS queue holding some ".json" messages in the queue. I want to pull the messages from amazon SQS via ES...

Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

When I simply run the following code, I always gets this error. s3 = boto3.resource('s3') bucket_name = "python-sdk-sample-%s" % uuid.uuid4() print("Creating new bucket with name:",...

How to choose an AWS profile when using boto3 to connect to CloudFront

I am using the Boto 3 python library, and want to connect to AWS CloudFront. I need to specify the correct AWS Profile (AWS Credentials), but looking at the official documentation, I see no way to...

When to use DataFrame.eval() versus pandas.eval() or Python eval()

I have a few dozen conditions (e.g., foo > bar) that I need to evaluate on ~1 MM rows of a DataFrame, and the most concise way of writing this is to store these conditions as a list of strings and...

How to upload crawled data from Scrapy to Amazon S3 as csv or json?

What are the steps to upload the crawled data from Scrapy to the Amazon s3 as a csv/jsonl/json file? All i could find from the internet was to upload scraped images to the s3 bucket. I'm currently...

Ansible ec2: "boto required for this module"

When I run this simple Ansible playbook: - name: EC2 Test Example hosts: localhost connection: local gather_facts: False tasks: - name: EC2 Instance ec2: # Amazon EC2 key pair...

Difference in boto3 between resource, client, and session?

I am using Python 2.7.12 in Ubuntu 16.04 LTS. I'm learning how to use boto3 from the following link: https://boto3.readthedocs.io/en/latest/guide/quickstart.html#using-boto-3. My doubt is when to...

Move/copy data from one folder to another on AWS S3

I am looking for all the methods for moving/copying the data from one folder to another on AWS S3 bucket. Method 1: Via AWS CLI (Most easy) Download and install awscli on ur instance, I am using...

Flask python app crashes on heroku

I tried pushing my code directly to Heroku for hosting but I got so many errors. Now, what I did was host the code on Github, and then link the git repo with Heroku. Now, the Heroku app was...

one hot encoding for frequent values only

I am looking to do one hot encoding to a column, but only for those that are very frequent. All that are below a threshold T will be put in their own category. My strategy was to create a...

Could not get AWS load balancer details using get_all_load_balancers in boto

I'm trying to get the load balancers which are created inside AWS. Below is my code elb_conn = boto.ec2.elb.connect_to_region(aws_access_key_id=AWSaccesskey,...

Load S3 Data into AWS SageMaker Notebook

I've just started to experiment with AWS SageMaker and would like to load data from an S3 bucket into a pandas dataframe in my SageMaker python jupyter notebook for analysis. I could use boto to...

Failed building wheel for xmlsec - mac

I'm installing my mac on python virtualenv environment. Whenever I install requirement packages command "pip install -r requirements.txt" i got this messages. pip install -r requirements.txt...

Cuda Runtime Error cudaErrorNoDevice: no CUDA-capable device is detected

I am using Chainer, Cupy for CUDA 8.0 . I am trying to train machine learning model using python3.5 script, but I got this error: cupy.cuda.runtime.CUDARuntimeError: cudaErrorNoDevice: no...

gsutil rsync from google cloud storage to s3 hangs on large file

I am experiencing a hang on a larger file (560MB) when attempting to backup my google cloud storage bucket to aws s3 bucket using the following command: gsutil -m rsync -rd gs://<MyGoogleBucket>/...

Ansible failed to complete because of deprecated Python even though Python 3 is installed

I'm trying to run LaunchKit from google to get some app screenshots. I've gone through all of the steps on the GitHub page (https://github.com/LaunchKit/LaunchKit) for the open source code. After...

How to programmatically set up Airflow 1.10 logging with localstack s3 endpoint?

In attempt to setup airflow logging to localstack s3 buckets, for local and kubernetes dev environments, I am following the airflow documentation for logging to s3. To give a little context,...

How to send a GraphQL query to AppSync from python?

How do we post a GraphQL request through AWS AppSync using boto? Ultimately I'm trying to mimic a mobile app accessing our stackless/cloudformation stack on AWS, but with python. Not javascript or...

Logging custom metrics using AWS Cloudwatch Agent and Python

We send up custom metrics to AWS using Python (see existing code below) and separately use the AWS CloudWatch Agent to send up metrics for our EC2 machine. However, we'd like to stop sending the...

AWS EMR pandas conflict with numpy in pyspark after bootstrapping

After launching cluster with the below bootstrap code and getting the below stdout, when I try to import pandas in pyspark, i get the following error due to conflict with a different numpy version...