如何获得Python的Mechanize来发布ajax请求? [英] How do I get Python's Mechanize to POST an ajax request?

问题描述

我要搜索的网站正在使用javascript:

The site I'm trying to spider is using the javascript:

request.open("POST", url, true);

要获取有关我需要抓取的ajax的其他信息.我尝试了以下各种排列方式:

To pull in extra information over ajax that I need to spider. I've tried various permutations of:

r = mechanize.urlopen("https://site.tld/dir/" + url, urllib.urlencode({'none' : 'none'}))

让Mechanize获取页面，但总是导致我再次获得登录HTML，这表明出了点问题.根据Firebug的说法，Firefox似乎没有向POST添加任何HTTP数据，并且我正在添加一个空字段来尝试强制urlopen使用"POST"而不是"GET"，以希望站点忽略该字段.我以为Mechanize的urlopen确实包含cookie.但是作为HTTPS，很难对事务进行调试以进行调试.

to get Mechanize to get the page but it always results in me getting the login HTML again, indicating that something is wrong. Firefox doesn't seem to add any HTTP data to the POST according to Firebug, and I'm adding an empty field to try and force the urlopen to use "POST" instead of "GET" hoping the site ignores the field. I thought that Mechanize's urlopen DOES include cookies. But being HTTPS it's hard to wireshark the transaction to debug.

有更好的方法吗?

似乎也没有用于Mechanize的不错的API文档，仅是示例.这很烦人.

Also there doesn't seem to be decent API documentation for Mechanize, just examples. This is annoying.

推荐答案

这是我想出的:

req = mechanize.Request("https://www.site.com/path/" + url, " ")
req.add_header("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.7) Gecko/20100713 Firefox/3.6.7")
req.add_header("Referer", "https://www.site.com/path")
cj.add_cookie_header(req)
res = mechanize.urlopen(req)

有趣的是机械化调用中的".请求将其强制为"POST"模式.显然，该网站并没有在一个空格处阻塞:)

Whats interesting is the " " in the call to mechanize.Request forces it into "POST" mode. Obviously the site didn't choke on a single space :)

它也需要cookie.我使用以下命令调试了标头:

It needed the cookies as well. I debugged the headers using:

hh = mechanize.HTTPHandler()
hsh = mechanize.HTTPSHandler()
hh.set_http_debuglevel(1)
hsh.set_http_debuglevel(1)
opener = mechanize.build_opener(hh, hsh)
logger = logging.getLogger()
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.NOTSET)
mechanize.install_opener(opener)

反对Firebug的显示.

Against what Firebug was showing.

这篇关于如何获得Python的Mechanize来发布ajax请求?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！