python调用CNBC后端API [英] Calling back-end API of CNBC in python
本文介绍了python调用CNBC后端API的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
作为
导入请求参数 = {"queryly_key": "31a35d40a9a64ab3","查询": "冠状病毒","endindex": "0",批量":100",打回来": "","showfaceted": "真实",时区偏移":-120","facetedfields": "格式","facetedkey": "格式|",面值":"!新闻稿|","needtoptickers": "1","附加索引": "4cd6f71fbf22424d,937d600b0d0d4e23,3bfbe40caee7443e,626fdfcd96444f28"}目标 = ["cn:title", "_pubDate", "cn:liveURL", "description"]定义主(网址):使用 requests.Session() 作为请求:对于页面,枚举中的项目(范围(0, 1100, 100)):打印(f提取第#页{页+1}")params["endindex"] = 项目r = req.get(url, params=params).json()r['results'] 中的 for 循环:打印([循环 [x] 为目标中的 x])main("https://api.queryly.com/cnbc/json.aspx")
Pandas
DataFrame
版本:
导入请求将熊猫导入为 pd参数 = {"queryly_key": "31a35d40a9a64ab3","查询": "冠状病毒","endindex": "0",批量":100",打回来": "","showfaceted": "真实",时区偏移":-120","facetedfields": "格式","facetedkey": "格式|",面值":"!新闻稿|","needtoptickers": "1","附加索引": "4cd6f71fbf22424d,937d600b0d0d4e23,3bfbe40caee7443e,626fdfcd96444f28"}目标 = ["cn:title", "_pubDate", "cn:liveURL", "description"]定义主(网址):使用 requests.Session() 作为请求:阿林 = []对于页面,枚举中的项目(范围(0, 1100, 100)):打印(f提取第#页{页+1}")params["endindex"] = 项目r = req.get(url, params=params).json()r['results'] 中的 for 循环:allin.append([loop[x] for x in目标])new = pd.DataFrame(allin, columns=["Title", "Date", "Url", "Description"])new.to_csv("data.csv", index=False)main("https://api.queryly.com/cnbc/json.aspx")
输出:
As a followup to this question, how can I locate the XHR request which is used to retrieve the data from the back-end API on CNBC News in order to be able to scrape this CNBC search query?
The end goal is to have a doc with: headline, date, full article and url.
I have found this: https://api.sail-personalize.com/v1/personalize/initialize?pageviews=1&isMobile=0&query=coronavirus&qsearchterm=coronavirus
Which tells me I don't have access. Is there a way to access information anyway?
解决方案
Actually my previous answer for you were addressing your question regarding the XHR
request:
But here we go with a screenshot
:
import requests
params = {
"queryly_key": "31a35d40a9a64ab3",
"query": "coronavirus",
"endindex": "0",
"batchsize": "100",
"callback": "",
"showfaceted": "true",
"timezoneoffset": "-120",
"facetedfields": "formats",
"facetedkey": "formats|",
"facetedvalue":
"!Press Release|",
"needtoptickers": "1",
"additionalindexes": "4cd6f71fbf22424d,937d600b0d0d4e23,3bfbe40caee7443e,626fdfcd96444f28"
}
goal = ["cn:title", "_pubDate", "cn:liveURL", "description"]
def main(url):
with requests.Session() as req:
for page, item in enumerate(range(0, 1100, 100)):
print(f"Extracting Page# {page +1}")
params["endindex"] = item
r = req.get(url, params=params).json()
for loop in r['results']:
print([loop[x] for x in goal])
main("https://api.queryly.com/cnbc/json.aspx")
Pandas
DataFrame
version:
import requests
import pandas as pd
params = {
"queryly_key": "31a35d40a9a64ab3",
"query": "coronavirus",
"endindex": "0",
"batchsize": "100",
"callback": "",
"showfaceted": "true",
"timezoneoffset": "-120",
"facetedfields": "formats",
"facetedkey": "formats|",
"facetedvalue":
"!Press Release|",
"needtoptickers": "1",
"additionalindexes": "4cd6f71fbf22424d,937d600b0d0d4e23,3bfbe40caee7443e,626fdfcd96444f28"
}
goal = ["cn:title", "_pubDate", "cn:liveURL", "description"]
def main(url):
with requests.Session() as req:
allin = []
for page, item in enumerate(range(0, 1100, 100)):
print(f"Extracting Page# {page +1}")
params["endindex"] = item
r = req.get(url, params=params).json()
for loop in r['results']:
allin.append([loop[x] for x in goal])
new = pd.DataFrame(
allin, columns=["Title", "Date", "Url", "Description"])
new.to_csv("data.csv", index=False)
main("https://api.queryly.com/cnbc/json.aspx")
Output: view online
这篇关于python调用CNBC后端API的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文