发送Scrapy中的请求 [英] Send Post Request in Scrapy

问题描述

我正在尝试从Google Play商店抓取最新评论，但我需要发出请求才能获得最新评价。

与邮递员申请作品

但是使用in terminal的帖子请求会给我服务器错误

例如：此页面 https://play.google.com/store/apps/details?id= com.supercell.boombeach

  curl -HContent-Type：application / json-X POST -d '{id：com.supercell.boombeach，reviewType：'0'，reviewSortOrder：'0'，pageNum：'0'} https://play.google.com/store / getreviews 
  

出现服务器错误和

Scrapy只是忽略此行：

  frmdata = {id：com.supercell.boombeach，reviewType： 0，reviewSortOrder：0，pageNum：0} 
 url =https://play.google.com/store/getreviews
 yield Request（url，callback = self.parse， method =POST，body = urllib.urlencode（frmdata））
  

解决方案>

确保 formdata 中的每个元素的类型为string / unicode

  frmdata = {id：com.supercell.boombeach，reviewType：'0'，reviewSortOrder：'0'，pageNum：'0'} 
 url = https://play.google.com/store/getreviews
 yield FormRequest（url，callback = self.parse，formdata = frmdata）
  

我认为这会做

 在[1] http import FormRequest 
 
 In [2]：frmdata = {id：com.supercell.boombeach，reviewType：'0'，reviewSortOrder：'0'，pageNum ：'0'} 
 
 In [3]：url =https://play.google.com/store/getreviews
 
在[4]：r = 2015-05-20 14：40：09 + 0530 [默认值] DEBUG：Crawled（200） < POST https://play.google.com/store/getreviews> （referer：None）
 [s]可用Scrapy对象：
 [s] crawler< scrapy.crawler.Crawler object at 0x7f3ea4258890> 
 [s] item {} 
 [s] r< POST https://play.google.com/store/getreviews> 
 [s] request< POST https://play.google.com/store/getreviews> 
 [s] response< 200 https://play.google.com/store/getreviews> 
 [s] settings< scrapy.settings.Settings object at 0x7f3eaa205450> 
 [s] spider< Spider'default'at 0x7f3ea3449cd0> 
 [s]有用的快捷方式：
 [s] shelp（）Shell帮助（打印帮助）
 [s] fetch（req_or_url） $ b [s] view（response）在浏览器中查看响应
  

I am trying to crawl latest reviews from google play store but I need to make a post request to get the latest reviews.

with the the postman post request works and I get desired response

but a post request using in terminal gives me server error

For ex: for this page https://play.google.com/store/apps/details?id=com.supercell.boombeach

curl -H "Content-Type: application/json" -X POST -d '{"id": "com.supercell.boombeach", "reviewType": '0', "reviewSortOrder": '0', "pageNum":'0'}' https://play.google.com/store/getreviews

gives a server error and

Scrapy just ignores this line:

frmdata = {"id": "com.supercell.boombeach", "reviewType": 0, "reviewSortOrder": 0, "pageNum":0}
        url = "https://play.google.com/store/getreviews"
        yield Request(url, callback=self.parse, method="POST", body=urllib.urlencode(frmdata))

解决方案

Make sure that each element in your formdata is of type string/unicode

frmdata = {"id": "com.supercell.boombeach", "reviewType": '0', "reviewSortOrder": '0', "pageNum":'0'}
url = "https://play.google.com/store/getreviews"
yield FormRequest(url, callback=self.parse, formdata=frmdata)

I think this will do

In [1]: from scrapy.http import FormRequest

In [2]: frmdata = {"id": "com.supercell.boombeach", "reviewType": '0', "reviewSortOrder": '0', "pageNum":'0'}

In [3]: url = "https://play.google.com/store/getreviews"

In [4]: r = FormRequest(url, formdata=frmdata)

In [5]: fetch(r)
 2015-05-20 14:40:09+0530 [default] DEBUG: Crawled (200) <POST      https://play.google.com/store/getreviews> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f3ea4258890>
[s]   item       {}
[s]   r          <POST https://play.google.com/store/getreviews>
[s]   request    <POST https://play.google.com/store/getreviews>
[s]   response   <200 https://play.google.com/store/getreviews>
[s]   settings   <scrapy.settings.Settings object at 0x7f3eaa205450>
[s]   spider     <Spider 'default' at 0x7f3ea3449cd0>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

这篇关于发送Scrapy中的请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！