How to percent-encode URL parameters in Python?

If I do url = "http://example.com?p=" + urllib.quote(query) It doesn't encode / to %2F (breaks OAuth normalization) It doesn't handle Unicode (it throws an exception) Is there a better library?

Get webpage contents with Python?

I'm using Python 3.1, if that helps. Anyways, I'm trying to get the contents of this webpage. I Googled for a little bit and tried different things, but they didn't work. I'm guessing that this...

How do I catch a specific HTTP error in Python?

I have import urllib2 try: urllib2.urlopen("some url") except urllib2.HTTPError: <whatever> but what I end up is catching any kind of HTTP error. I want to catch only if the specified...

How to urlencode a querystring in Python?

I am trying to urlencode this string before I submit. queryString = 'eventName=' + evt.fields["eventName"] + '&' + 'eventDescription=' + evt.fields["eventDescription"];

Downloading and unzipping a .zip file without writing to disk

I have managed to get my first python script to work which downloads a list of .ZIP files from a URL and then proceeds to extract the ZIP files and writes them to disk. I am now at a loss to...

can we use XPath with BeautifulSoup?

I am using BeautifulSoup to scrape an URL and I had the following code import urllib import urllib2 from BeautifulSoup import BeautifulSoup url = ...

Python: Need to wait before BeautifulSoup and Urllib can parse a website

I am trying to get the current world population in real-time but when the webpage first loads up it takes a couple seconds to retrieve the data. When i run the program i get loading... instead of...

Web scraping without knowledge of page structure

I'm trying to teach myself a concept by writing a script. Basically, I'm trying to write a Python script that, given a few keywords, will crawl web pages until it finds the data I need. For...

Open web in new tab Selenium + Python

So I am trying to open websites on new tabs inside my WebDriver. I want to do this, because opening a new WebDriver for each website takes about 3.5secs using PhantomJS, I want more speed... I'm...

Reading a specific class in a webpage using python

I have a script which reads data from a webpage using HTMLParser: import urllib from HTMLParser import HTMLParser import re class get_HTML_Info(HTMLParser): def handle_data(self, data): ...

Python - make a POST request using Python 3 urllib

I am trying to make a POST request to the following page: http://search.cpsa.ca/PhysicianSearch In order to simulate clicking the 'Search' button without filling out any of the form, which adds...

Putting latitudes and longitudes into a distance matrix, google map API in python

I have a pandas dataframe in the following format:[dataframe.][1] [![enter image description here][1]][1] Now I want to put all the pairs of Start Latitude and Start Longitude into origins and the...

formatting data from a new source in pyalgotrade

I am trying to adapt a pyalgotrade feed to use data streamed from another source. I am getting an error inside of the run_strategy method when I try to point the feed to data data obtained using...

PDFminer: PDFTextExtractionNotAllowed Error

I'm trying to extract text from pdfs I've scraped off the internet, but when I attempt to download them I get the error: File "/usr/local/lib/python2.7/dist-packages/pdfminer/pdfpage.py", line...

Trying to load data from google storage bucket results in a apiclient.errors.HttpError: <HttpError 403... returned "Forbidden">

so right now im trying to write a python script which loads a few files from our google storage. I'm using Windows, have installed the Google Cloud SDK and set everything up with gsutil so i can...

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

I am trying to use the betbrain.py from Github (https://github.com/gto76/betbrain-scraper) that has the following code: #!/usr/bin/python3 # # Usage: betbrain.py [URL or FILE] [OUTPUT-FILE] #...

Best Practices with Anaconda and Brew

I have just got a new Macbook with OSX Sierra, so want to ensure my development environment is setup properly. I am looking to follow the 'best practices' mentioned here:...

Using swiftclient to stream large files straight to Openstack

I need to upload files from a file storage service to OSS, alongside a Rabbit MQ queue. Currently, I am downloading files using urllib, and then using uploading them to OSS using python...

Recursion Depth Exceeded, pickle and BeautifulSoup

I want to pickle html from websites. I save the html to a list and try to pickle it. An example of one such list is the html from brckhmptn.com/tour. Of course the html from this site is a lot, is...

Python Profanity Checker Error

The profanity checker seems to be working but does not return either (response: false) or response: true) Here is a look at the code: import urllib.request def read_text(): quotes =...

Python selector.select returning empty socket as readable

I'm having problems implementing a basic HTTPServer in Python 3.6 on windows 10 Pro x64. Essentially the selector being used in the base implementation of socketserver.serve_forever() uses a...

Find final redirected url in Python

import requests def extractlink(): with open('extractlink.txt', 'r') as g: print("opened extractlink.txt for reading") contents = g.read() headers =...

"Insert Into" statement causing errors due to "Parameter 7 (""): The supplied value is not a valid instance of data type float."

I'm loading a batch of CSV files into a SQL Server table using Python one row at a time. The files each contain a number of free text fields and erroneous data which I trim and rename before...

Error calling CF API login one time passcode

I am working with the CF API RESTful services. Trying to get an access token from cloud foundry's UAA API using https://login..../oauth/token web method. I have verified that headers & body...

Max retries exceeded with url: / Caused by ProxyError

i wanted to get some proxy list from this webPage; https://free-proxy-list.net/ but i stuck in this error and dont know how to fix it. requests.exceptions.ProxyError:...

Parsing a waf log to get the host ip to prevent dos attack

I'm trying to create a report where if a ip request have attacked the server more then 1000 in one minute it is an dos attack. Aws waf is logging the logs in s3 and using lambda we will check if...

Python's requests triggers Cloudflare's security while urllib does not

I'm working on an automated web scraper for a Restaurant website, but I'm having an issue. The said website uses Cloudflare's anti-bot security, which I would like to bypass, not the...

What does this error say? [CS50 WEB FINANCE]

I am doing CS50 Web track finance buy and this is my code which gives the error given below The error which it is giving is confusing and I dont understand what it is saying as well as how to...

I am not able to use RemBg Python Software. Help me

I was using RemBg Software provided in this Repository. Whenever I used it. It gives me the following error- Failed to import ahead-of-time-compiled modules. This is expected on first...

How to use NordVPN in Python?

I was wondering how to use NordVPN servers to change IP address during my web automation with Python. Let's say to use it with mechanize this way: br.open("url", proxies="nordvpn proxy") Thanks...