new .net 4 multi-threading technique?

i sat in on part of a Microsoft PDC and heard the presenter talk about the cool new way that .net 4 and VS2010 allow for multi-threading. the code is smaller and cleaner and simpler, the logic is...

asp.net/ C# screen scaping done easily?

whats the simplest way to do screen scraping using c# and .net 4.0? are their libraries i can reuse? i think i heard of an html tool pack for this but can not find it now...

BeautifulSoup: How do I extract all the <li>s from a list of <ul>s that contains some nested <ul>s?

My source code looks like: <h3>Header3 (Start here)</h3> <ul> <li>List items</li> <li>Etc...</li> </ul> <h3>Header 3</h3> <ul> <li>List items</li> <ul> <li>Nested list...

WebClient.DownloadString() returns string with peculiar characters

I have an issue with some content that we are downloading from the web for a screen scraping tool that I am building. in the code below, the string returned from the web client download string...

Can scrapy be used to scrape dynamic content from websites that are using AJAX?

I have recently been learning Python and am dipping my hand into building a web-scraper. It's nothing fancy at all; its only purpose is to get the data off of a betting website and have this data...

Scraping data from all asp.net pages with AJAX pagination implemented

I want to scrap a webpage containing a list of user with addresses, email etc. webpage contain list of user with pagination i.e. page contains 10 users when I click on page 2 link it will load...

how to set cookies in curl using LIB_http

I'm receiving cookies message again & again while scraping the page manta.com message is Array ( [FILE] => Oops. Before you can move on, please activate your browser cookies. I am using cookies...

Updating value in iterrow for pandas

I am doing some geocoding work that I used selenium to screen scrape the x-y coordinate I need for address of a location, I imported an xls file to panda dataframe and want to use explicit loop to...

Enter value into aspx form in R and webscraping result

I am trying to webscrape property information using a county website First what I would like to webscrape: URL: http://reparcelasmt.loudoun.gov/search/commonsearch.aspx?mode=parid For example:...

Web-scraping of mobile apps?

Is there any program/library available that can scrape the contents of an mobile apps' screen? The goal is to have a nice data structure for the Instagram "Following" feed.

Rvest html_table error - Error in out[j + k, ] : subscript out of bounds

I'm getting an error message that I can't make sense of. My code: url <- "https://en.wikipedia.org/wiki/California_State_Legislature,_2017%E2%80%9318_session" leg <- read_html(url) testdata <-...

Puppeteer waitForSelector on multiple selectors

I have Puppeteer controlling a website with a lookup form that can either return a result or a "No records found" message. How can I tell which was returned? waitForSelector seems to wait for...

Web scraping for product details, not a list / table in UIPath

I have situation where I want to scrape a profile of a company for example with 20 / 30 different attributes layout on one page, save each of those elements as a column title, and paginate through...

Scraping multiple links by scrapy

I scraped a web page & all the useful links I stored in a list & now I want to scrape those links which are in the list. So how may I do it?

Can a website detect when using Chromium via Puppeteer?

When scraping a website using Chromium with Node plus Puppeteer (not Selenium and ChromeDriver), it is able to detect and blocks me throwing customized error instead of serving the pages, while...

Scrapy not scraping all results

I want to scrape the NHL match data. Then, two rows must be scraped per a match(a url). When I scrape one match(one url), result is not empty. But when I scrape ten matches, result is empty,...

Scraping info page

I am trying to scrape the data from for example this link: https://i.instagram.com/api/v1/users/6862425230/info/ Here is my code: import requests from bs4 import BeautifulSoup url =...

Web scraping for Google Ads using python and beautifulsoup

I am trying to scrape for Google search results that have the "Ad" in the right, ie scraping for Google ad links from search results. I have the following script, where I am stuck at...

VBA WebScraping returning nothing to excel

I've been trying to scrape data from a WebSite, as my previous question indicates. I was able to figure what my problem was thanks to the comunity, but now I'm facing another problem. I don't get...

VBA WebScraping return empty values

I have the following code to scrape data from a website, the problem is that it isn't scraping any data, it doesn't show any errors but doesn't give me any results as well... Option...

Scraping .aspx site after click

I am attempting to scrape scheduling data for my squadron from: https://www.cnatra.navy.mil/scheds/schedule_data.aspx?sq=vt-9 I have figured out how to extract the data using BeautifulSoup...

How can I use Selenium to handle and select elements from the date picker on the website?

I have a problem in Selenium to scrape the data that I want by selecting the specific date from the date picker on the website. However, the code below I tried (e.g. I pick 11 April 2019) could...

Selenium scroll down slowly

I'm trying to do dynamic web scraping on a javascript-rendered webpage using Python. However, the elements only load when I scroll down the page slowly. I have...

How to fix Newspaper3k 403 Client Error for certain URL's?

I am trying to get a list of articles using a combo of the googlesearch and newspaper3k python packages. When using article.parse, I end up getting an error: newspaper.article.ArticleException:...

"INVALID" is not a valid start token

Can't for the life of me figure out why I'm getting this error for one of the applications I'm trying to scrape. I have the following prometheus.yml: # prometheus.yml global: scrape_interval:...

extract data out of a windows application with python

So I have been doing some web scraping with python. I now want to scrape data out of a very old dictionary software on windows. I want to be able to extract meaning of words and save it in a...

What tool for scraping in Laravel

I would like build a website needing scraping in other site, in Laravel 7. I have to fill forms and click. I saw that it's possible with python. The scraping will do itself every hours. I saw also...

Selenium: access denied

I am trying to scrape some data from LV website with Selenium and keep getting 'Access Denied' screen once 'sign in' button clicked. I feel like there is a protection against this because all...

Python Requests Library - Scraping separate JSON and HTML responses from POST request

I'm new to web scraping, programming, and StackOverflow, so I'll try to phrase things as clearly as I can. I'm using the Python requests library to try to scrape some info from a local movie...

TweepError: Twitter error response: status code = 403

I'm trying to extract the amount of #btc since 2019-01-01 per day. I know the error is about permission, but I'm already using the keys generated from Twitter developer's portal. Here's my code, I...