如何使用urllib2/httplib调用Twitter的流/过滤源? [英] How to call Twitter's Streaming/Filter Feed with urllib2/httplib?

问题描述

更新:

当我尝试用切实可行的尼克的答案提出的解决方案时，我将其从回答中切换回去，并切换到Google的urlfetch:

I switched this back from answered as I tried the solution posed in cogent Nick's answer and switched to Google's urlfetch:

logging.debug("starting urlfetch for http://%s%s" % (self.host, self.url))
result = urlfetch.fetch("http://%s%s" % (self.host, self.url), payload=self.body, method="POST", headers=self.headers, allow_truncated=True, deadline=5)
logging.debug("finished urlfetch")

但不幸的是，完成的urlfetch 从未被打印-我看到日志中发生了超时(5秒后返回200)，但是执行似乎没有返回.

but unfortunately finished urlfetch is never printed - I see the timeout happen in the logs (it returns 200 after 5 seconds), but execution doesn't seem tor return.

大家好

我正在尝试使用Twitter的流式(又称流水线)API a>使用Google App Engine(我知道这可能不是一个长期的游戏，因为您无法使用GAE永久保持连接的打开)，但是到目前为止，我并没有运气让我的程序真正生效解析Twitter返回的结果.

I'm attempting to play around with Twitter's Streaming (aka firehose) API with Google App Engine (I'm aware this probably isn't a great long term play as you can't keep the connection perpetually open with GAE), but so far I haven't had any luck getting my program to actually parse the results returned by Twitter.

某些代码:

logging.debug("firing up urllib2")
req = urllib2.Request(url="http://%s%s" % (self.host, self.url), data=self.body, headers=self.headers)
logging.debug("called urlopen for %s %s, about to call urlopen" % (self.host, self.url))
fobj = urllib2.urlopen(req)
logging.debug("called urlopen")

不幸的是，执行此操作时，我的调试输出从不显示打印的称为urlopen 行.我怀疑正在发生的事情是Twitter保持了连接打开，而urllib2没有返回，因为服务器没有终止连接.

When this executes, unfortunately, my debug output never shows the called urlopen line printed. I suspect what's happening is that Twitter keeps the connection open and urllib2 doesn't return because the server doesn't terminate the connection.

Wireshark显示请求已正确发送，并返回响应并返回结果.

Wireshark shows the request being sent properly and a response returned with results.

我尝试将 Connection:close 添加到我的请求标头中，但是没有成功.

I tried adding Connection: close to my request header, but that didn't yield a successful result.

关于如何使它起作用的任何想法?

Any ideas on how to get this to work?

推荐答案

urllib是

urllib on App Engine is a thin wrapper around the urlfetch API. You're right about what's happening: Twitter's streaming API never terminates its response, so it times out, and urlfetch throws an exception.

如果直接使用urlfetch，则可以设置超时(最长10秒)，并将allow_truncated设置为True，这样可以获得部分结果.但是，Twitter流API确实不是App Engine的理想选择，因为App Engine请求的执行时间限制为30秒，而urlfetch请求不能逐步发送结果，也不能花费超过10秒的时间.使用Twitter的标准" API将是更好的选择.

If you use urlfetch directly, you can set the timeout (up to 10 seconds), and set allow_truncated to True so you can get the partial result. The Twitter streaming API really isn't a good match for App Engine, though, because App Engine requests are limited to 30 seconds of execution time, and urlfetch requests can't send back results progressively, or take more than 10 seconds. Using Twitter's 'standard' API would be a better option.

这篇关于如何使用urllib2/httplib调用Twitter的流/过滤源?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！