Parsing meta tags efficiently with lxml?

I'm parsing HTML pages with lxml. The pages have meta tags as follows: <meta property="og:locality" content="Detroit" /> <meta property="og:country-name" content="USA" /> How can I use lxml to...

Stop pip from failing on single package when installing with requirements.txt

I am installing packages from requirements.txt pip install -r requirements.txt The requirements.txt file reads: Pillow lxml cssselect jieba beautifulsoup nltk lxml is the only package failing...

How to change default install location for pip

I'm trying to install Pandas using pip, but I'm having a bit of trouble. I just ran sudo pip install pandas which successfully downloaded pandas. However, it did not get downloaded to the...

python pip trouble installing from requirements.txt

I've had great luck with pip in the past, but working at installing some stuff in a venv on is giving me some headaches. I keep getting errors like No distributions at all found for...

How to install cffi package on AWS Beanstalk

This question looks the same as this post, but since there was no answer, I am re-asking here. I have a Django project to be deployed on AWS Beanstalk, which is using a package cffi. When I run eb...

How to clean up /Library/Python/2.7/site-packages under Mac OS X El Capitan

So I was messing around following all kinds of tutorials telling me to sudo pip install instead of doing it right by using a virtualenv for properly handling what my individual web apps need. The...

Where is pip cache folder?

Where is Python pip cache folder? I had an error during install and now reinstall packages using cache files. Where is that directory? I want to backup them for install in the future. Is it...

jQuery/cheerio selector, context and root - what's the difference?

I'm new to Javascript and would like to use the library Cheerio to do some webscraping. Came across this text in the introduction to the library. Am not sure what the difference is between a...

import error after clean install of fiona

I installed fiona as follows: conda install -c conda-forge fiona It installed without any errors. When I try to import fiona, I get the following error: Traceback (most recent call last): File...

Unable to scrape data from Expedia.com

I am scraping Data from Expedia using spyder and It was working on my local system now. Initially, it was showing the same issue with expedia.com then I switched to expedia.co.in.It showing this...

scrapy - not able to upload data to s3

I am using scrapy to scrape the data from one website which is working fine but i am not able to upload the scraped data onto amazon s3 Looking at the scrapy documentation this is what I have in...

Could not install packages due to an EnvironmentError: [Errno 2] No Such file or directory

this is what I'm using to install packages, the only one that works is requests pip._internal.main(['install', 'requests']) pip._internal.main(['install', 'lxml']) pip._internal.main(['install',...

Could not find a version that satisfies the requirement ItsDangerous==1.0.0. Django, Pythonanywhere

When I try to deploy my application to pythonanywhere, the following error it is returned. Could not find a version that satisfies the requirement ItsDangerous==1.0.0 How can I solve it? I...

Scrapy 1.6 : DNS lookup failed

I am new to Scrapy and im trying to crawl this website https://www.timeanddate.com/weather/india and its throwing DNS lookup error. The code i wrote for scraping works perfectly in shell so my...

How can I get more information on Python unexpected SIGABRT?

I'm using: MacOS Catalina, version 10.15 (19A603). python 3.7.4 pip3 Running and Debugging the following Python code within venv: import jose print(jose) from jose import jwt token =...

CentOS Python, mod_wsgi is not working properly

I want to deploy Djanogo Celery application, and I encountered error ModuleNotFoundError: No module named Xyz but when I go to folder where is manage.py is and run python3 manage.py runserver I...

Ansible failed to complete because of deprecated Python even though Python 3 is installed

I'm trying to run LaunchKit from google to get some app screenshots. I've gone through all of the steps on the GitHub page (https://github.com/LaunchKit/LaunchKit) for the open source code. After...

Scrapy 404 Error - FormRequest redirecting problem on BrickSeek website

I am currently trying to login brickseek's website using FormRequest method but I am unable to login successfully. I keep on getting 404 error when using the scrapy crawl command. It seems to me...

Error "Running as root without --no-sandbox is not supported"

I try to implement scrapy-puppeteer library for my project (https://pypi.org/project/scrapy-puppeteer/) I implement PuppeteerMiddleware according to documentation from library Here is code which I...

scrapy does not scrape parent URL

I have an html file demo1.html with code: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Title</title> </head> <body> “What is this obsession people have with...

Scrapy - TypeError: can only concatenate str (not "list") to str

While I try to gather a list of URL from a website and put them to combine with a base URL, then continue it inside the page. Once combine and will crawl those Url 1 by 1 then crawl the details of...

How to ask ACCESS_SURFACE_FLINGER permission on Android with Kivy/Buildozer?

Hi there and first of all a big thanks to all of you who helped me without knowing it. For a noob like me, stack overflow is really precious. I'm new to coding, learned Python3 to see if I can,...

Scrapy python - I keep getting Crawled 0 pages

I have tried to follow multiple tutorials but no matter what I try I'm always getting the same result "Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)" my code is very...

raise AttributeError: Response content isn't text Scarpy proxy pool. How to solve?

Raises AttributeError: Response content isn't text. I am using scrapy_proxy_pool and scrapy_user_agents. I am trying to find each and every link of target website. import scrapy class...

Alpine ERROR: unsatisfiable constraints: py3-pandas (missing):

I have the following dockerfile: FROM alpine:latest ADD crontab.txt /crontab.txt ADD script.sh /script.sh COPY entry.sh /entry.sh ADD app /app RUN chmod 755 /script.sh /entry.sh RUN...

I get 'AttributeError: module 'sipbuild.api' has no attribute 'prepare_metadata_for_build_wheel' when trying to build a Docker image of a python app

I am trying to build a docker image of a python application. It fails with the following error: "Installing build dependencies: finished with status 'done' Getting requirements to build wheel:...

Heroku SSL connection error unsupported protocol

I have been using Heroku for a while to host my Discord bot. It has been connecting to a MySQL database hosted on ClearDB successfully. However, very recently, whenever I use the bot and it tries...

Install Scrapy on Windows Server 2019, running in a Docker container

I want to install Scrapy on Windows Server 2019, running in a Docker container (please see here and here for the history of my installation). On my local Windows 10 machine I can run my Scrapy...

pip3.6 install mysqlclient==1.3.12 fails with error: unknown type name ‘my_bool’; did you mean ‘bool

I have a project that worked on ubuntu 16.04 with python 3.6 but now we are trying to make it run on ubuntu 20.04 with same python version. I need to install all requirements on the venv and...

Can't Install Taurus on Windows 10 with Python 3.10.0

Can't Install Taurus on Windows 10 with Python 3.10.0. Following Prerequisites are installed Get Python 3.7+ from http://www.python.org/downloads and install it, don't forget to enable "Add...