刮除启用JS的网站以实现自动化 [英] Scraping JS enabled sites for automation

问题描述

我正在尝试抓取一个特定网站( https://chartink.com/screener/test-121377 )我正在尝试访问URL中提供的表.该表可以是动态形式.该网站启用了JS，因此尝试HTML抓取对我来说是多余的.接下来，我尝试使用cURL访问XHR元素，它确实起作用.我使用下面给出的代码:

I am trying to scrape one particular site (https://chartink.com/screener/test-121377) I am trying to access the provided table in the URL. This table can be of dynamic form. The site is JS enabled hence it was redundant of me to try HTML scraping. Next, I tried to access XHR elements using cURL, it did work. I used given below code:

import requests
headers = {
'authority': 'chartink.com',
'pragma': 'no-cache',
'cache-control': 'no-cache',
'accept': 'application/json, text/javascript, */*; q=0.01',
'x-csrf-token': 'HYMZ6XnOySvn1w4fHEN0Hu1ublftvidgdBD5qywR',
'x-requested-with': 'XMLHttpRequest',
'user-agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36',
'sentry-trace': 'd9b8f7ffcae445f7a74913d5ac143918-9fa7556ebb76186c-0',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'origin': 'https://chartink.com',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
'referer': 'https://chartink.com/screener/test-121377',
'accept-language': 'en-IN,en;q=0.9,hi-IN;q=0.8,hi;q=0.7,en-GB;q=0.6,en-US;q=0.5',
'cookie': '_ga=GA1.2.468738165.1600680012; remember_web_59ba36addc2b2f9401580f014c7f58ea4e30989d=eyJpdiI6Inp6YWdERUJHSGxoWmhRaE00MGhONHc9PSIsInZhbHVlIjoiSXN1d09PXC9USmpGb2lFR21naFd1Q3RlcCtKSDdUXC8zdGRnMkRoQmNcL2loXC9tT0tMXC9sc0JtallJVCtNczVhVjhIdGUyb0xqMTIydlc0RWtQYVZcLzF1aFd3aXRPM0M3MDVObUtNK1FtdmJtZjY1YUxNbTlyTHhNWFhjU3VZNXpUTWY0SjZ4M1o1VEcydk1DWDY5YWVSdWRjbVwvQmkyYmlDT2VMWFVCM043Y2JIY21UTjNsdnJLaE1qSnNybmM2WVhISyIsIm1hYyI6IjhlNjBiZWUxMWU3NTBkOTRjM2NlYTZiNGI3YzJhYjg4M2E0OWI2MDU3ZDZkYWUzNzE1YjYxOTRhMTllMjY2ZWQifQ%3D%3D; __utmc=102564947; PHPSESSID=e3797d9e58f9f56b0307e7840f6839f7; bfp_sn_rf_8b2087b102c9e3e5ffed1c1478ed8b78=Direct/External; bfp_sn_rt_8b2087b102c9e3e5ffed1c1478ed8b78=1601299430046; bafp=d8afd2b0-018d-11eb-b966-a1985446bf25; hbcm_sd=1%7C1602315380999; _gid=GA1.2.729526333.1602491131; __utma=102564947.468738165.1600680012.1601134509.1602499714.2; __utmz=102564947.1602499714.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); XSRF-TOKEN=eyJpdiI6InVXSEFpN2xiYWZqWDkzNEdWZVBTSkE9PSIsInZhbHVlIjoiQlM2eVBYMjdYR3N4QTh6YmtFWmNoTUtOVHBIOFVoUG04WTdcL05iUm4yZExQR3ZucHlPVHhucVwvbjlHTW9VcmZCIiwibWFjIjoiMWEyMzViZDQ5YWI1YzIzY2UzYTdkNzRlN2UyZmYzNmY1ZWNkYTBmZmJmOWI2YjZmOGRmN2U2Y2IzOGE5ZmQwNiJ9; ci_session=eyJpdiI6ImtcLzNBOGxOb1l2bFlyOEFnQ2dxVHNBPT0iLCJ2YWx1ZSI6IjRpNVhXOWtGZlV0TWdwZlhIVGRmWFgrcGFyblwveGlkdVU5aHJraVdvbFROOURUa2Zhc2pcLzJNOE9qa0FwcjlwbSIsIm1hYyI6IjVlYTUwNDgxN2U0NTU4OWE0NTgxOTMwNjNjMzdhM2Q5MjRiMjJjMTBkYzI0ZTdkOTg1YTg5YmIxNTM3NzFlMDQifQ%3D%3D',}
data = {'scan_clause': '( {33489} ( latest close > 10 ) ) '}
response = requests.post('https://chartink.com/screener/process', headers=headers, data=data)
data = json.loads(response.text)
df = pd.json_normalize(data['data'])

有效.但是，其中的某些元素将在一段时间后过期.CSRF令牌，岗哨和cookie.

It worked. However certain elements of it will get expire after a certain time. CSRF token, sentry- trace and cookies.

一段时间后，由于令牌不匹配，代码在运行时将开始出现错误.

After a certain time, the code will start having errors while running it due to tokens mismatch.

然后我使用下面给定的方法(跟随另一个线程)，但是它也不起作用.

I then used the given below method (following another thread) but it also doesn't work.

import requests_html
session = requests_html.HTMLSession()
r = session.get('https://chartink.com/screener/test-121377')
r.html.render(sleep=5)
items = r.html.find("table#DataTables_Table_0",first=True)
for item in items.find("tr"):
data = [td.text for td in item.find("th,td")]
print(data)

但是我又收到错误:

RuntimeError: Cannot use HTMLSession within an existing event loop. Use AsyncHTMLSession instead.

当我使用AsyncHTMLSession时，还有一些其他错误.总而言之，这对我来说是死胡同.

When I used AsyncHTMLSession then there were some other errors. All in all, it's been a dead-end for me.

任何一种方法都可以添加一个解释.

One can add an explanation in either approach.

推荐答案

您可以一次性获得全部137个结果.尝试使用基本url收集csrf令牌，然后发送http post请求以及适当的参数以获取json响应，以便从中获取所需的内容.这是您可以如何做:

You can get all 137 results in one go. Try using the base url to collect csrf token and then send http post requests along with appropriate parameters to get json response in order to fetch required content out of it. Here is how you can:

import requests
from bs4 import BeautifulSoup

link = "https://chartink.com/screener/test-121377"
url = 'https://chartink.com/screener/process'

payload = {
    'scan_clause': '( {33489} ( latest close > 10 ) )'
}

with requests.Session() as s:
    r = s.get(link)
    soup = BeautifulSoup(r.text,"html.parser")
    csrf = soup.select_one("[name='csrf-token']")['content']
    s.headers['x-csrf-token'] = csrf
    r = s.post(url,data=payload)
    for item in r.json()['data']:
        print(item['name'],item['nsecode'],item['per_chg'],item['close'],item['volume'])

结果如下:

Mrf Limited MRF -0.34 58587.1 9057
BANKNIFTY BANKNIFTY -0.56 23712.8 0
Shree Cements Limited SHREECEM -1.14 20956.1 56254
Page Industries Limited PAGEIND -0.29 20898.75 26311
Nestle India Limited NESTLEIND -0.82 15902.2 58171
Bosch Limited BOSCHLTD -1.64 12529.1 37951

这篇关于刮除启用JS的网站以实现自动化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！