Python BigQuery真的很奇怪的超时 [英] Python BigQuery really strange timeout

问题描述

我正在构建一项服务，将数据流式传输到bigquery中。下面的代码完美地工作，如果我删除需要4-5分钟加载（我正在preaching一些映射）的部分

  from googleapiclient导入发现
来自oauth2client导入文件$ b $来自oauth2client导入客户$ b $来自oauth2client导入工具
 $ b来自oauth2client.client import SignedJwtAssertionCredentials 
 
＃ ＃加载电子邮件和密钥
 credentials = SignedJwtAssertionCredentials（email，key，scope ='https：//www.googleapis.com/auth/bigquery'）
 
如果凭证是None或凭证。无效：
 raw_input（'invalid key'）
 exit（0）
 
 http = httplib2.Http（）
 http = credentials.authorize（http）
 
 service = discovery.build（'bigquery'，'v2'，http = http）
 
 
 ##这不会挂起，因为它在长操作之前
 service.tabledata（）。insertAll（...）
 
 
 ##一些需要5分钟才能执行的代码
r = load_mappings（）
 ## ak长操作
 
 ##这会挂起
 service.tabledata（）。insertAll（...）
  

如果我离开需要5分钟执行的部分，Google API将停止响应我之后执行的请求。它只是挂在那里，甚至不会返回错误。我甚至在10-20分钟内看到会发生什么，它只是坐在那里。如果我按ctrl + c，我得到这个：

$ $ p code> ^ CTraceback（最近调用最后一次）：
文件 ./to_bigquery.py，第116行，在< module>
main（sys.argv）
文件./to_bigquery.py，第101行，在
print service.tabledata（）。insertAll（projectId = p_n，datasetId =XXX ，tableId =％s_XXXX％str（shop），body = _mybody）.execute（）
文件/usr/local/lib/python2.7/dist-packages/oauth2client/util.py，行132，在locations_wrapper
中返回包装（* args，** kwargs）
文件/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py，第716行，在执行
body = self.body，headers = self.headers）
文件/usr/local/lib/python2.7/dist-packages/oauth2client/util.py，第132行，位于positions_wrapper
返回包装（*参数，** kwargs）
文件/usr/local/lib/python2.7/dist-packages/oauth2client/client.py，第490行，在new_request中
redirections，connection_type）
请求
（response，content）= self的文件/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py，第1593行。 _request（conn，authority，uri，request_uri，method，body，headers，redirectio ns，cachekey）
文件/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py，第1335行，在_request
（response，content）= self._conn_request （conn，request_uri，method，body，headers）
文件/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py，第1291行，在_conn_request
response = conn.getresponse（）
文件/usr/lib/python2.7/httplib.py，第1030行，getresponse
response.begin（）
文件/ usr / lib / python2.7 / httplib.py，第407行，开头为
version，status，reason = self._read_status（）
文件/usr/lib/python2.7/httplib.py，行365，在_read_status
行= self.fp.readline（）
文件/usr/lib/python2.7/socket.py，行430，在readline
data = recv（ 1）
文件/usr/lib/python2.7/ssl.py，第241行，在recv
返回self.read（buflen）
文件/ usr / lib / python2 .7 / ssl.py，第160行，读入
返回self._sslobj.read（len）

我已管理通过在凭证授权之前进行大的加载操作来临时修复它，但对我来说这似乎是一个错误。我错过了什么？

编辑：我在等待的时候遇到了一个错误：

 文件/usr/local/lib/python2.7/dist-packages/oauth2client/util.py，第132行，位置位置wrapper 
返回包装（* args，** kwargs） 
文件/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py，第716行，执行
 body = self.body，headers = self.headers）
文件/usr/local/lib/python2.7/dist-packages/oauth2client/util.py，第132行，位于位置包装程序
返回包装（* args，** kwargs）
文件/usr/local/lib/python2.7/dist-packages/oauth2client/client.py，第490行，在new_request 
 redirections，connection_type）
文件/ usr / local / lib / python2.7 / dist-packages / httplib2 / __init__.py，第1593行，请求
（response，content）= self._request（conn，authority，uri，request_uri，method，body，headers，redirections， cachekey）
文件/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py，行1335，在_request 
（response，content）= self._conn_request（conn，request_uri，method，body，headers）
文件/usr/local/lib/python2.7/dist-packages/httplib2 /__init__.py，第1291行，在_conn_request 
 response = conn.getresponse（）
文件/usr/lib/python2.7/httplib.py，第1030行，在getresponse 
 response.begin（）
文件/usr/lib/python2.7/httplib.py，行407，开头
 version，status，reason = self._read_status（）
文件/usr/lib/python2.7/httplib.py，行365，在_read_status 
行= self.fp.readline（）
文件/usr/lib/python2.7/socket .py，第430行，在readline 
 data = recv（1）
文件/usr/lib/python2.7/ssl.py，第241行，在recv 
 return self .read（buflen）
读取
的文件/usr/lib/python2.7/ssl.py，第160行，返回self._sslobj.read（len）
 socket.error ：[Errno 110]连接超时
  

它表示超时。这似乎发生在冷桌子上。

解决方案

  def refresh_bq（self）：
凭证= SignedJwtAssertionCredentials（email，key，scope ='https：//www.googleapis.com/auth/bigquery'）
 
如果凭证是None或credentials.invalid：
 raw_input（'invalid key'）
 exit（0）
 
 http = httplib2.Http（）
 http = credentials.authorize（http）
 
 service = discovery.build（'bigquery'，'v2'，http = http）
 self.service = service 
  

我每次运行self.refresh_bq（）时都会执行一些不需要预处理的插入操作，并且工作起来非常完美。凌乱的黑客攻击，但我需要尽快完成工作。有def。一个错误的地方。

I am building a service to stream data into bigquery. The following code works flawlessly if i remove the part that takes 4-5 minutes to load (i am precaching some mappings)

from googleapiclient import discovery
from oauth2client import file
from oauth2client import client
from oauth2client import tools

from oauth2client.client import SignedJwtAssertionCredentials

## load email and key
credentials = SignedJwtAssertionCredentials(email, key, scope='https://www.googleapis.com/auth/bigquery')

if credentials is None or credentials.invalid:
        raw_input('invalid key')
        exit(0)

http = httplib2.Http()
http = credentials.authorize(http)

service = discovery.build('bigquery', 'v2', http=http)


## this does not hang, because it is before the long operation
service.tabledata().insertAll(...)


## some code that takes 5 minutes to execute
r = load_mappings()
## aka long operation

## this hangs
service.tabledata().insertAll(...)

If i leave the part that takes 5 minutes to execute, the Google API stops responding to the requests i do afterwards. It simply hangs in there and doesn't even return an error. I left it even 10-20 minutes to see what happens and it just sits there. If i hit ctrl+c, i get this:

^CTraceback (most recent call last):
  File "./to_bigquery.py", line 116, in <module>
    main(sys.argv)
  File "./to_bigquery.py", line 101, in main
    print service.tabledata().insertAll(projectId=p_n, datasetId="XXX", tableId="%s_XXXX" % str(shop), body=_mybody).execute()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 716, in execute
    body=self.body, headers=self.headers)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 490, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1593, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1335, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1291, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()   
  File "/usr/lib/python2.7/socket.py", line 430, in readline
    data = recv(1)
  File "/usr/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)

I have managed to temporarily fix it by placing the big loading operation BEFORE the credentials authorization, but it seems like a bug to me. What am i missing?

EDIT: I have managed to get an error, while waiting:

  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 716, in execute
    body=self.body, headers=self.headers)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 490, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1593, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1335, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1291, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.7/socket.py", line 430, in readline
    data = recv(1)
  File "/usr/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
socket.error: [Errno 110] Connection timed out

It said timeout. This seems to happen with cold tables..

解决方案

def refresh_bq(self):
    credentials = SignedJwtAssertionCredentials(email, key, scope='https://www.googleapis.com/auth/bigquery')

    if credentials is None or credentials.invalid:
        raw_input('invalid key')
        exit(0)

    http = httplib2.Http()
    http = credentials.authorize(http)

    service = discovery.build('bigquery', 'v2', http=http)
    self.service = service

i am running self.refresh_bq() everytime i do some inserts that do not require preprocessing, and it works flawlessly. messy hack, but i needed to make it work ASAP. There is def. a bug somewhere.

这篇关于Python BigQuery真的很奇怪的超时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！