如何使用urllib2 / httplib调用Twitter的Streaming / Filter Feed？ [英] How to call Twitter's Streaming/Filter Feed with urllib2/httplib?

问题描述

更新：

当我试着在Nick的答案中提出的解决方案并切换到Google的urlfetch时，我将其转回了答案：

  logging.debug（启动urlfetch为http：//％s％s％（self.host，self.url））
 result = urlfetch.fetch（http：//％s％s％（self.host，self.url），payload = self.body，method =POST，headers = self.headers，allow_truncated = True，截止日期= 5 ）
 logging.debug（finished urlfetch）
  

但不幸完成urlfetch 永远不会被打印 - 我看到日志中发生超时（它在5秒后返回200），但执行似乎不会返回。

---

全部 -

我正在尝试使用Twitter的使用Google App Engine流式传输（又名firehose）API （我知道这可能不是很长长期打一场你不能保持与GAE一直开放的连接），但到目前为止，我还没有运气让我的程序实际解析Twitter返回的结果。

一些代码：

  logging.debug（点燃urllib2）
 req = urllib2.Request（url = ％s％s％（self.host，self.url），data = self.body，headers = self.headers）
 logging.debug（为％s％s调用urlopen， （self.host，self.url））
 fobj = urllib2.urlopen（req）
 logging.debug（called urlopen）
 命名为urlopen 的行。   
 
 
 Wireshark显示正确发送的请求，结果返回的响应。
 
 
 
我尝试在我的请求标题中添加 Connection：close ，但没有获得成功的结果。
 
 
 
关于如何让这个工作起作用的任何想法？     div> 
 App Engine上的urllib是 urlfetch API <一>。对于发生的事情你是正确的：Twitter的流媒体API永远不会终止它的响应，所以它会超时，并且urlfetch会抛出一个异常。
 
 
 
如果你直接使用urlfetch，你可以设置超时（最多10秒），并将allow_truncated设置为True，以便获得部分结果。不过，Twitter流API实际上与App Engine不太匹配，因为App Engine请求被限制在30秒的执行时间，urlfetch请求无法逐步发回结果，或者需要超过10秒。使用Twitter的'标准'API是更好的选择。
 
Update: 

I switched this back from answered as I tried the solution posed in cogent Nick's answer and switched to Google's urlfetch:
logging.debug("starting urlfetch for http://%s%s" % (self.host, self.url))
result = urlfetch.fetch("http://%s%s" % (self.host, self.url), payload=self.body, method="POST", headers=self.headers, allow_truncated=True, deadline=5)
logging.debug("finished urlfetch")
but unfortunately finished urlfetch is never printed - I see the timeout happen in the logs (it returns 200 after 5 seconds), but execution doesn't seem tor return.



Hi All-

I'm attempting to play around with Twitter's Streaming (aka firehose) API with Google App Engine (I'm aware this probably isn't a great long term play as you can't keep the connection perpetually open with GAE), but so far I haven't had any luck getting my program to actually parse the results returned by Twitter.

Some code:
logging.debug("firing up urllib2")
req = urllib2.Request(url="http://%s%s" % (self.host, self.url), data=self.body, headers=self.headers)
logging.debug("called urlopen for %s %s, about to call urlopen" % (self.host, self.url))
fobj = urllib2.urlopen(req)
logging.debug("called urlopen")
When this executes, unfortunately, my debug output never shows the called urlopen line printed. I suspect what's happening is that Twitter keeps the connection open and urllib2 doesn't return because the server doesn't terminate the connection.

Wireshark shows the request being sent properly and a response returned with results.

I tried adding Connection: close to my request header, but that didn't yield a successful result.

Any ideas on how to get this to work?
解决方案
urllib on App Engine is a thin wrapper around the urlfetch API. You're right about what's happening: Twitter's streaming API never terminates its response, so it times out, and urlfetch throws an exception.

If you use urlfetch directly, you can set the timeout (up to 10 seconds), and set allow_truncated to True so you can get the partial result. The Twitter streaming API really isn't a good match for App Engine, though, because App Engine requests are limited to 30 seconds of execution time, and urlfetch requests can't send back results progressively, or take more than 10 seconds. Using Twitter's 'standard' API would be a better option.

                        
这篇关于如何使用urllib2 / httplib调用Twitter的Streaming / Filter Feed？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！