How to use -pip- to install packages that can work with Anaconda?

I am trying to install some additional packages that do not come with Anaconda. All of these packages can be installed using pip install PackageName. However, when I type this command at the...

Can MechanicalSoup (python 3x) deal with javascript in a website?

I could successfully login and navigate through a website but when I inspected the logout button it is like javascript:__doPostBack('ctl00$lnkBtnLogout','') as link, follow_link() doesn't work on...

Error: Could not find suitable distribution for Requirement.parse('pytest-runner')

I tried to install MechanicalSoup package via pip but i got these errors : Download error on https://pypi.org/simple/pytest-runner/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert...

How to call a function within a class?

I need to call a function print3() which is inside the class two. But I need to call from inside of the class but not being in any function: class two: y = 2 def __init__(self): ...

Sending a click with requests_html and pyppeteer python

I'm attempting to login to a website, click a button, and then scrape some data. The page must be rendered because it's all in JavaScript (and thus unavailable if you were [for example] to View...

Filling <select> with no 'name' attribute in MechanicalSoup

I am quite new to webscraping so I am having a trouble automatically filling in forms. Below is the relevant python code and html source code of the webpage. So, what I am trying to do is to fill...

Mechanicalsoup Catpcha issue on submit Python

I am trying to create a bot in python using mechanicalsoup to scrap a trademark status details from - http://ipindiaonline.gov.in/eregister/Application_View.aspx This site is made up on asp.net...

Site returns login page again when scraping after logging in successfully once using MechanicalSoup?

Im trying to scrape some data from Twitter using BeautifulSoup as a part of a project. To scrape the ‘following’ section I need to first login, so I tried doing so using MechanicalSoup. I know...

How to open up browser after login with MechanicalSoup and Python script?

I write a Python script with the help of MechanicalSoup to automate a login task. The script successfully runs and I can see that get_url() method returns a url which tells login is successful. At...

Why is my Travis CI build failing with "File Not Found" errors when those files exist?

I'm trying to determine why my Travis CI build job is failing with "File Not Found" errors like the one below, despite the files having absolute paths and almost certainly existing in the...

Excluding HTML-Elements (recursive=False not working)

The idea is, to check the last 3 pages of a german medical-news-page. On each of these pages are 5 with links to separate articles. The program checks, if the "href" of each is already existing...

MechanicalSoup StatefulBrowser: Unable to Open URL

I have a Python script using MechanicalSoup StatefulBrowser to open URL that used to work. But it stopped working recently opening a specific website, and I haven't changed any code. I tried...

Mechanize uploading image returns 403 error

i am trying to upload a image to ebay-kleinanzeigen.de with mechanize, but everytime i try it, i am getting a 403 error. The code for the login: import mechanicalsoup browser =...

Send binary image by POST request

i am trying to do a post request to upload a img to https://pasteboard.co/, but i am always getting a 500 response which tells me, there is a missing file. The file is really existing and the path...

mysql in python with visual studio code

I installed python 2.7.15 version on my computer (windows 10 64 bit) and installed mysql using pip install mysql-connector and from command prompt i can see mysql-connector working fine and i can...

Logging in using Requests in Python

Hello I am trying to login to https://www.neighborwho.com using Requests for Python but with my code the website response keeps telling me that it cannot find any user with my username when in...

How can I use the CSS selector to select a form when there are multiple forms on a page?

I am trying to fill out a specific form on a webpage but the CSS selector I am using returns an error I have tried these separately: 1. browser.select_form('form[method="post"]') 2....

How to add a cookie to existing cookies in MechanicalSoup

I know MechanicalSoup has a function called set_cookiejar() but it replaces the current cookiejar completely. I want to know how to add new cookies to existing cookies.

Python: Remain logged in a website using mechanicalsoup

I am a newbie in Python. I want to write a script in python where I run thousands of URLs and save the response. In order to access these URLs, credentials are required. So, I have written a basic...

requests.exceptions.ConnectionError: Failed to establish a new connection: 0x05: Connection refused

I know there are already alot of topics writing about this topic, but none could held me. import mechanicalsoup browser = mechanicalsoup.StatefulBrowser() browser.set_user_agent( 'Mozilla/5.0...

Web Scrape Vanguard Investment Account (login)

I am trying to automatically scrape my balance info from my Vanguard investment account. I am stuck trying to login to the account. I've tried requests and mechanicalsoup and so far I have not...

Using Mechanicalsoup to navigate multiple pages / forms

I've had success using mechanicalsoup with single pages / single forms, but am having difficulty with a multistep problem. The pages I am attempting to navigate start here:...

MechanicalSoup with Login: Unable to browser.download_link() with given URL & save-path

Im currently trying to fetch some files from an internal network. I managed to fetch the URLs of some files in the format https://example.site/files/testfile.pdf. Now I've tried to download this...

mechanicalsoup TypeError: 'module' object is not callable

I've read the previous posts here on the subject problem, although not specifically for mechanicalsoup, which mostly indicated a possible conflict between the module and class name, but I don't...

Python: how to fill out a web form and get the resulting page source

I am trying to write a python script that will scrape http://www.fakenewsai.com/ and tell me whether or not a news article is fake news. I want the script to input a given news article into the...

certificate verify failed with MechanicalSoup

I want to retrieve some data from my router using MechanicalSoup. However when I try I get "certificate verify failed". As I see it I have two options Find a way to disable SSL verification Tell...

How to add proxy support to mechanicalsoup?

import mechanicalsoup proxies = { "http": "socks5://127.0.0.1:49260", "https": "socks5://127.0.0.1:49260" } url = 'https://whatismyip.com/' browser =...

Mechanicalsoup freezes with some websites

I use MechanicalSoup for a custom search engine. When I try to open some websites, MechanicalSoup freezes, up to few hours, and gives an exception only after this delay. Sometimes it works, but it...

login to website with onclick button to submit

I am trying to use python3.7 for web-scraping a website that needs user_email and password to login. The login form of the login page is as follows: <div id="login"> <form...

Log In and Web Scrape with Python 3 but action='#' and possibly Java script

I am trying to us Python 3 to scrape my data from Ancestry.com using Beautifulsoup and Mechanicalsoup but I am running into a few issues trying to log in. Here is the form's HTML on...