"SSL certificate verify failed" using pip to install packages

I am trying to install the Scrapy package (among others) for python using pip. I have tried doing the installation using python 3 and python 2, I have installed/upgraded the setuptools like so: $...

select2 disable / enabled not working

I have a select element with select2 plugin. Version is 4.0. It works well but i cant change disabled option dynamically. $('#list1').select2({ theme: "bootstrap", disabled: true, ...

Unable to scrape data from Expedia.com

I am scraping Data from Expedia using spyder and It was working on my local system now. Initially, it was showing the same issue with expedia.com then I switched to expedia.co.in.It showing this...

scrapy - not able to upload data to s3

I am using scrapy to scrape the data from one website which is working fine but i am not able to upload the scraped data onto amazon s3 Looking at the scrapy documentation this is what I have in...

Python cannot be push to Heroku

I try to deploy a Django app to Heroku and the push get rejected. The result shows that : Push rejected, failed to compile Python app. From the error script, it seems that the model "ConfigParser'...

Scrapy: ImportError: No module named scrapy_proxies

I have installed scrapy_proxies with pip install scrapy_proxies. But whenever I run my spider I get the following error log: scrapy crawl event -o items_new.csv 2018-09-13 01:15:19...

Scrapy 1.6 : DNS lookup failed

I am new to Scrapy and im trying to crawl this website https://www.timeanddate.com/weather/india and its throwing DNS lookup error. The code i wrote for scraping works perfectly in shell so my...

Scrapy spider not saving items to PostgreSQL database

I have some Scrapy spiders that get properties advertisement info and stores on database. It was already working when I start in the company, but we had to migrate our DB from GCP to AWS, so I've...

CentOS Python, mod_wsgi is not working properly

I want to deploy Djanogo Celery application, and I encountered error ModuleNotFoundError: No module named Xyz but when I go to folder where is manage.py is and run python3 manage.py runserver I...

Scrapy 404 Error - FormRequest redirecting problem on BrickSeek website

I am currently trying to login brickseek's website using FormRequest method but I am unable to login successfully. I keep on getting 404 error when using the scrapy crawl command. It seems to me...

Error "Running as root without --no-sandbox is not supported"

I try to implement scrapy-puppeteer library for my project (https://pypi.org/project/scrapy-puppeteer/) I implement PuppeteerMiddleware according to documentation from library Here is code which I...

apscheduler+scrapy+asyncio Can't execute first task smoothly

version: python 3.7、Scrapy 2.1.0、APScheduler 3.6.1 i create a simple spider for test # -*- coding: utf-8 -*- import scrapy class TestSpider(scrapy.Spider): name = 'test' start_urls =...

Linkedin scraper to extract skills

I'm trying to scrape people's public profiles to get most common skills for certain roles. I'm able to extract email, company, name, position etc. but I can't get the skills. I'm using Selector...

scrapy does not scrape parent URL

I have an html file demo1.html with code: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Title</title> </head> <body> “What is this obsession people have with...

Scrapy splash not load content

I started using selenium a few months ago, then scrapy. Learning tutorials from Udemy, youtube, and stackoverflow questions, all the scrapes were successful, until I started working with this page...

Scrapy user login not working with FormRequest.from_response()

I just set up an simple Scrapy-Spider to crawl some data protected by an userlogin. For some hours I tried to login with Scrapy FormRequest.from_response() into the following website by using the...

How to scrape data from a dynamic website with Selenium

I am new to selenium and want to scrape price and offer end time from a Udemy Course link. How can i do this? The price and course end time is dynamically loaded to the website. I know how to...

Scrapy with splash settings works in scrapy shell, fails otherwise

I'm trying to scrape the content from this link on my macOS, using scrapy with scrapy_splash settings and BeautifulSoup I followed the instructions in the documentation I tested every single...

Possible bug in scrapy 2.3.0 Invalid syntax async=False

I keep getting syntax error when I'm trying to run scrapy in AWS ubuntu 18.04 instance: scrapy crawl pcz -o px.csv here's the log [email protected]:~/free_proxy/free_proxy$ scrapy crawl...

AttributeError: 'NoneType' object has no attribute 'strip' - Scrapy doesn't crawl all the elements

My spider doesn't crawl all the elements. As I can see now, one of the errors is an attribute error which I don't know how to fix it. This is a non-English website that I want its numbers to be...

Cannot get Python3 to recognize installed modules

I have scoured across StackOverflow and Google and have not been able to find a solution. I'm currently running the macOS Big Sur beta and I have Python 3.8.5 installed via homebrew. I have pip3...

Selenium driver on google next button, NoSuchElementException

I am writing a script that deals with google. I have successfully searched for what I wanted using the selenium web driver however I would like to navigate to the next page of results. my code...

Scrapy hidden memory leak

Background - TLDR: I have a memory leak in my project Spent a few days looking through the memory leak docs with scrapy and can't find the problem. I'm developing a medium size scrapy project,...

Is there a way to hide overriden credentials settings from Scrapy output?

When overriding default settings values like FTP_PASSWORD or MAIL_PASS they're automatically displayed on console output. 2020-10-01 14:20:45 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot:...

Scrapy - TypeError: can only concatenate str (not "list") to str

While I try to gather a list of URL from a website and put them to combine with a base URL, then continue it inside the page. Once combine and will crawl those Url 1 by 1 then crawl the details of...

How to ask ACCESS_SURFACE_FLINGER permission on Android with Kivy/Buildozer?

Hi there and first of all a big thanks to all of you who helped me without knowing it. For a noob like me, stack overflow is really precious. I'm new to coding, learned Python3 to see if I can,...

Scrapy python - I keep getting Crawled 0 pages

I have tried to follow multiple tutorials but no matter what I try I'm always getting the same result "Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)" my code is very...

how to use scrapy-rotating-proxies with full settings or rotate ip/per request?

hello folks, I am scraping a website and using scrapy-rotating-proxies, however i also tried other proxies but they are not suited my requirements or i can't implement them as i...

raise AttributeError: Response content isn't text Scarpy proxy pool. How to solve?

Raises AttributeError: Response content isn't text. I am using scrapy_proxy_pool and scrapy_user_agents. I am trying to find each and every link of target website. import scrapy class...

Install Scrapy on Windows Server 2019, running in a Docker container

I want to install Scrapy on Windows Server 2019, running in a Docker container (please see here and here for the history of my installation). On my local Windows 10 machine I can run my Scrapy...