how do i use node.js with request or anything to get data over https through a http proxy

I have i instance of polipo running on a liunx server on port 8123 I need to make get and post requests with https inside of node through this proxy server. how would i do this. btw proxychains is...

How to run multiple Tor processes at once with different exit IPs?

I am brand new to Tor and I feel like multiple Tors should be considered. The multiple tors I mentioned here are not only multiple instances, but also using different proxy ports for each, like...

How to connect to https site with Scrapy via Polipo over TOR?

Not entirely sure what the problem is here. Running Python 2.7.3, and Scrapy 0.16.5 I've created a very simple Scrapy spider to test connecting to my local Polipo proxy so I can send requests out...

Node.js + Polipo https request error

I'm doing a simple example using polipo with tor and it's working fine on my firefox browser. I've set polipo as the proxy and the browser works just fine. Then I tried doing a simple request in...

scrapy has response status 400 , but browser response is ok?

I have this strange situation, I have a link that works on all borwsers that I currently have (chrome,IE,firefox), I tried to crawl the page using scrapy in python. however I get response.status...

rtorrent through a proxy

It's not that much of a question, rather a confirmation that what I did is right or not and if it is safe or not. Until now what I have found googling around is that you cannot run rtorrent...

Twisted Python Failure - Scrapy Issues

I am trying to use SCRAPY to scrape this website's search reqults for any search query - http://www.bewakoof.com . The website uses AJAX (in the form of XHR) to display the search results. I...

Scrapy with TOR (Windows)

I created a Scrapy project with several spiders to crawl some websites. Now I want to use TOR to: Hide my ip from the crawled servers; Associate my requests to different ips, simulating accesses...

missing polipo/config file in /usr/local/etc/

I use brew to install polipo through Mac OS terminal. It seems successfully install, but I can not find the config file and edit it.Can anyone help me figure out the reason?

Python script to convert adblock list in a forbidden file for Polipo proxy (regex)

On my system I currently running Polipo proxy, mainly for adblock purposes. By a search on internet I've found many shell scripts to convert adblock plus lists in Polipo forbidden file format;...

Configure proxy on mac with polipo

I am trying to configure proxy on my mac via polipo. I've installed polipo 1.1.1, but stuck to configure it. The point is that I can not find where the config file is to add proxy host and proxy...

How to prevent getting blacklisted while scraping Amazon

I try to scrape Amazon by Scrapy. but i have this error DEBUG: Retrying <GET...

check tor connection is established before running scrapy

I'd like to check tor before I start crawling using python scrapy. I am using polipo/tor/scrapy on linux. with this settup scrapy correctly using tor on its crawls. The way I check if the scrapy...

How to find out where a C struct is declared?

I am trying to add logging to the web proxy Polipo and as part of this need to log the url request in the following function [following line][1]: httpClientRequest(HTTPRequestPtr request, AtomPtr...

Scrapy-Splash with Tor

I have succeed to run Scrapy with Tor using this link: http://pkmishra.github.io/blog/2013/03/18/how-to-run-scrapy-with-TOR-and-multiple-browser-agents-part-1-mac/ But i couldn't run Splash with...

Scrapy-Splash running Tor Proxy

I try to build proxy settings of splash. Here I assign Tor or Polipo port address to set_proxy, but it doesn't work. I get 504 error: function main(splash) local host = "localhost" ...

Problems with cookies in scrapy - IP blocked

I'm new to scrapy (it is my first time), and I'm facing some problems. The whole idea is to scrap data from a website and store it on a Sqlite database, using python. I got blocked out while...

use polipo to convert shadowsocks into an HTTP proxy

My ssserver is started by docker image oddrationale/docker-shadowsocks: docker run -d -p 1984:1984 oddrationale/docker-shadowsocks -s 0.0.0.0 -p 1984 -k paaassswwword -m aes-256-cfb Then I use...

Polipo Proxy specify network interface to listen on

Right at the moment I am taking a closer look to the polipo proxy software. I already managed to setup an easy socks5 proxy with basic user authentication. All it needs to be finished is to change...

Scrapy and Tor/Privoxy unable to crawl [Connection refused 61]

I have the following code refered from here in my middlewares.py which i'm trying to change my ip in TOR with every request def _set_new_ip(): with Controller.from_port(port=9051) as...

Scrapy spider stops abruptly

I am using this example here. To change my identity with Tor/Privoxy but i have faced several issues such as having to type "scrapy crawl something.py" multiple times to start the spider or having...

How to configure polipo's HTTP proxy for docker to deploy Kubernetes

I am a beginner of Docker. Wanted to know if it's good practice to do that and what would be the best way to do that? System: Ubuntu LTS 16.04.2 I want to deploy Kubernetes on my server with...

Wrap tor socks proxy with http proxy in command line

I have installed tor via apt and it is listening on port number 9050 # netstat -ntlup | grep tor tcp 0 0 127.0.0.1:9050 0.0.0.0:* LISTEN 13566/tor ...

How to use Crawlera with selenium (Python, Chrome, Windows) without Polipo

So basically i am trying to use the Crawlera Proxy from scrapinghub with selenium chrome on windows using python. I checked the documentation and they suggested using Polipo like this: adding the...

Scrapy with Tor, Polipo in (Ubuntu) : how to renew IP

I have added the following code to /etc/polipo/config file: # This file only needs to list configuration variables that deviate # from the default values. See...

Crawlera proxies not working with python selenium and Chrome Web driver

I have just bought a plan from crawlera for proxies and according to their documentation using polipo is not working for me and according to polipo site, it is outdated already. How Can I use...

How to set different IP according to different commands of one single scrapy.Spider?

I have a bunch of pages to scrape, about 200 000. I usually use Tor and Polipo proxy to hide my spiders behaviors even if they are polite, we never know. So if I login this is useless to use one...

Bash, Waiting for tor proccess circuit to be complete "Bootstrapped 100%: Done" before follow other instructions

The main idea is the following: ###Launch tor & polipo ./launchTorPolipo.sh start 1 5 #wait the circuits to be complete #then ./otherScriptDependingOnTorPolipoStates.sh in the script...

How can proxy scrapy requests with Socks5?

Question: How can proxy scrapy requests with socks5? I know I can use polipo to convert Socks Proxy To Http Proxy But: I want to set a Middleware or some changes in scrapy.Request import...

Squid proxy server refuse connection for twitter

I have installed Squid proxy on my Ubuntu 16 vps. This is my squid.conf: auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/passwords auth_param basic realm proxy acl...