使用请求库用python模拟ajax请求 [英] Simulating ajax request with python using requests lib

问题描述

为什么 request 不下载此网页的响应?

Why does request not download a response for this webpage?

#!/usr/bin/python

import requests

headers={ 'content-type':'application/x-www-form-urlencoded; charset=UTF-8',
     'Accept-Encoding': 'gzip, deflate',
     'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0',
     'Referer' : 'http://sportsbeta.ladbrokes.com/football',
    }

payload={'N': '4294966750',
     'facetCount_156%23327': '12',
     'facetCount_157%23325': '8',
     'form-trigger':'moreId',
     'moreId':'156%23327',
     'pageId':'p_football_home_page',
     'pageType':'EventClass',
     'type':'ajaxrequest'
     }

url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'

r = requests.post(url, data=payload, headers=headers)

这些是我在 Firebug 中看到的 POST 的参数，收到的响应包含一个列表(足球联赛)，但是当我像这样运行我的 python 脚本时，我什么也没得到.

These are the parameters of the POST that I see in Firebug, and there the response received back contains a list (of football leagues), yet when I run my python script like this I get nothing.

(您可以在 Firefox 中通过点击 link 并查看 Firebug 中的 XHR.Firebug 响应按预期显示 HTML 正文.)

(you can see the request in Firefox by clicking the See All in the competitions section of the left hand nav bar of link and looking at the XHR in Firebug. The Firebug response shows the HTML body as expected.)

有人有什么想法吗?我对有效载荷中 % 符号的处理是否会造成任何问题?

Anyone any ideas? Will my handling of the % symbols in the payload be causing any trouble at all?

尝试使用会话

from requests import Request, Session

#turn post string into dict: 
def parsePOSTstring(POSTstr):
    paramList = POSTstr.split('&')
    paramDict = dict([param.split('=') for param in paramList])
    return paramDict

headers={'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0',
     'Referer' : 'http://sportsbeta.ladbrokes.com/football'
    }

#prep the data (POSTstr copied from Firebug raw source)
POSTstr = "moreId=156%23327&facetCount_156%23327=12&event=&N=4294966750&pageType=EventClass&
          pageId=p_football_home_page&type=ajaxrequest&eventIDNav=&removedSelectionNav=&
          currentSelectedId=&form-trigger=moreId"
payload = parsePOSTstring(POSTstr)

#end url
url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'

#start a session to manage cookies, and visit football page first so referer agrees
s = Session()
s.get('http://sportsbeta.ladbrokes.com/football')
#now visit disired url with headers/data
r = s.post(url, data=payload, headers=headers)

#print output
print r.text #this is empty

工作卷曲

curl 'http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'
-H 'Cookie: JSESSIONID=DE93158F07E02DD3CC1CC32B1AA24A9E.ecomprodsw015;
    geoCode=FRA; 
    FLAGS=en|en|uk|default|ODDS|0|GBP;
    ECOM_BETA_SPORTS=1;
    PLAYED=4%7C0%7C0%7C0%7C0%7C0%7C0'
-H 'Referer: http://sportsbeta.ladbrokes.com/football'
-H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) 
    Gecko/20100101 Firefox/27.0'  
--data 'facetCount_157%23325=8&moreId=156%23327&
        facetCount_156%23327=12&event=&
        N=4294966750&
        pageType=EventClass&pageId=p_football_home_page&
        type=ajaxrequest&eventIDNav=&
        removedSelectionNav=&currentSelectedId=&
        form-trigger=moreId' --compressed

然而这个卷曲有效.

推荐答案

这是我能想到的最小的工作示例:

Here's the smallest working example that I can come up with:

from requests import Session

session = Session()

# HEAD requests ask for *just* the headers, which is all you need to grab the
# session cookie
session.head('http://sportsbeta.ladbrokes.com/football')

response = session.post(
    url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController',
    data={
        'N': '4294966750',
        'form-trigger': 'moreId',
        'moreId': '156#327',
        'pageType': 'EventClass'
    },
    headers={
        'Referer': 'http://sportsbeta.ladbrokes.com/football'
    }
)

print response.text

您只是没有正确解码百分比编码的 POST 数据，所以 # 在实际的 POST 数据中被表示为 %23(例如 156%23327 应该是 156#327).

You just weren't decoding the percent-encoded POST data properly, so # was being represented as %23 in the actual POST data (e.g. 156%23327 should've been 156#327).

这篇关于使用请求库用python模拟ajax请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！