Python,从网站提取XHR响应数据 [英] Python, extract XHR response data from website
本文介绍了Python,从网站提取XHR响应数据的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试从 https:/中提取一些数据/www.barchart.com/stocks/signals/top-bottom/top?viewName=main .
我能够使用xpath方法从普通html提取数据,但是我注意到该网站从网络获取数据.
I am able to extract data from normal html using the xpath method, however i noticed that this website gets its data from a network.
我找到了我想要的数据的位置(barchart网站上的表格),如下图所示.
I have found the location of where the data I want is (the table from the barchart website) which is shown in the picture below.
我如何仅抓取响应部分?
How can i scrape just the response portion?
谢谢!
推荐答案
尝试以下代码.
from urllib import urlencode
headers = {
'cookie': 'XSRF-TOKEN=eyJpdiI6Ikk4cFdwcGlqMVVPRHM4MFc5Vk1ROWc9PSIsInZhbHVlIjoiUHRHbjZwSTZFTUkrTFRNZVJXczZsV2xZZnN1WEhmM1puakN5WWFsM0RUODhJRUZJYVA3XC9GZ1QyOUR5R0VqaXRmdDJIR0wyV2lBdXlFcTlxXC9HWFVqZz09IiwibWFjIjoiNTlkODY2MGM0YzJlZDQxMTI5ZmIwMmE0NWI5YzkzY2Q4NDg3MjhiODg4NDAxN2Q5NmYzYTE3MmUxZWQzZjk4MiJ9; laravel_session=eyJpdiI6ImFob1k3NWw1U2pBcWhKU3JLdEpDQkE9PSIsInZhbHVlIjoidnNNcXpKSHRmQkYyMGp2NTF4eUhTVVwvSmtidnAwMDV5eGdUWWVhZ2syTUlXaVdQV1JEUldYcG5lXC9mdUtnRkl2akNMR2ZKMkY2NFlWYTFOM2NPRm1uQT09IiwibWFjIjoiYTJhODkwMTJjZjI3NzJiOTE1YWY5MWJkYTNiYTNlMjZjNjI1YjgzZmJlYzhkMTU4Mjk1OWQ5MWEzMGU5OTM5NSJ9; _ga=GA1.2.674295962.1536391451; _gid=GA1.2.1388999250.1536391451; _gat_UA-2009749-51=1; Markets09122018PageView=1; Markets09122018WebinarClosed=true',
'dnt': '1',
'x-xsrf-token': 'eyJpdiI6Ikk4cFdwcGlqMVVPRHM4MFc5Vk1ROWc9PSIsInZhbHVlIjoiUHRHbjZwSTZFTUkrTFRNZVJXczZsV2xZZnN1WEhmM1puakN5WWFsM0RUODhJRUZJYVA3XC9GZ1QyOUR5R0VqaXRmdDJIR0wyV2lBdXlFcTlxXC9HWFVqZz09IiwibWFjIjoiNTlkODY2MGM0YzJlZDQxMTI5ZmIwMmE0NWI5YzkzY2Q4NDg3MjhiODg4NDAxN2Q5NmYzYTE3MmUxZWQzZjk4MiJ9',
'accept-language': 'en-US,en;q=0.9',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36',
'accept': 'application/json',
'referer': 'https://www.barchart.com/stocks/signals/top-bottom/top?viewName=main',
'authority': 'www.barchart.com',
'accept-encoding': 'gzip, deflate, br',
}
params = (
'lists': 'stocks.signals.top.current.us',
'orderDir': 'asc',
'fields': 'symbol,symbolName,lastPrice,priceChange,percentChange,opinion,opinionPrevious,opinionLastWeek,opinionLastMonth,symbolCode,symbolType,hasOptions',
'meta': 'field.shortName,field.type,field.description',
'hasOptions': 'true',
'page': '1',
'limit': '100',
'raw': '1',
)
yield Request('https://www.barchart.com/proxies/core-api/v1/quotes/get?' + urlencode(params) , headers=headers)
这篇关于Python,从网站提取XHR响应数据的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文