DeadlineExceededError：已超出响应HTTP请求的总体截止日期 [英] DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded

问题描述

我有一个cron作业，它调用vendor API来获取公司列表。一旦数据被提取，我们将这些数据存储到云数据存储中，如下面的代码所示。出于某种原因，在我触发cron作业的最后两天，开始看到错误。当我在本地调试代码时，我没有看到这个错误。

I have a cron job which calls vendor api to fetch the companies list. Once the data is fetched, we are storing that data into cloud datastore as shown in the below code . For some reason for last two days when i trigger the cron job , started seeing the errors. When i debug the code locally i dont see this error

    company_list = cron.rest_client.load(config, "companies", '')

    if not company_list:
        logging.info("Company list is empty")
        return "Ok"

    for row in company_list:
        company_repository.save(row,original_data_source, 
                                 actual_data_source)

存储库代码

  def save( dto, org_ds , act_dp):
   try:
    key = 'FIN/%s' % (dto['ticker'])
    company = CompanyInfo(id=key)
    company.stock_code = key
    company.ticker = dto['ticker']
    company.name = dto['name']
    company.original_data_source = org_ds
    company.actual_data_provider = act_dp
    company.put()
    return company
  except Exception:
    logging.exception("company_repository: error occurred saving the 
                       company record ")
    raise

错误

  DeadlineExceededError: The overall deadline for responding to the 
                          HTTP request was exceeded.

异常详情

Exception details

  Traceback (most recent call last):
  File   

"/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/googl
    e/appengine/runtime/wsgi.py", line 267, in Handle
    result = handler(dict(self._environ), self._StartResponse)
   File "/base/data/home/apps/p~svasti-173418/internal-
  api:20170808t160537.403249868819304873/lib/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/base/data/home/apps/p~svasti-173418/internal-
   api:20170808t160537.403249868819304873/lib/flask/app.py", line 1817, in 
    wsgi_app
      response = self.full_dispatch_request()
    File "/base/data/home/apps/p~svasti-173418/internal-
   api:20170808t160537.403249868819304873/lib/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/base/data/home/apps/p~svasti-173418/internal-api:20170808t160537.403249868819304873/lib/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/base/data/home/apps/p~svasti-173418/internal-api:20170808t160537.403249868819304873/internal/cron/company_list.py", line 21, in run
    company_repository.save(row,original_data_source, actual_data_source)
  File "/base/data/home/apps/p~svasti-173418/internal-api:20170808t160537.403249868819304873/internal/repository/company_repository.py", line 13, in save
    company.put()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3458, in _put
    return self._put_async(**ctx_options).get_result()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
    self.check_success()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 378, in check_success
    self.wait()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 362, in wait
    if not ev.run1():
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 268, in run1
    delay = self.run0()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 248, in run0
    _logging_debug('rpc: %s.%s', rpc.service, rpc.method)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 453, in service
    @property
DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded.

推荐答案

您的公司名单变得更大吗？

Has your company list been getting bigger?

您试图放置多少个实体？

How many entities are you trying to put?

尝试将它们保存为批处理，而不是按循环顺序保存。从 def save（dto，org_ds，act_dp）：中删除​​ company.put（）并使用 ndb.put_multi（）代替。

Try saving them as a batch, instead of sequentially in a loop. Remove company.put() from def save( dto, org_ds , act_dp): and use ndb.put_multi() afterwards instead.

company_list = cron.rest_client.load(config, "companies", '')

if not company_list:
    logging.info("Company list is empty")
    return "Ok"

company_objs=[]
for row in company_list:
    company_objs.append(company_repository.save(row,original_data_source, 
                             actual_data_source))
    # put 500 at a time
    if len(company_objs) > 500:
        ndb.put_multi(company_objs)
        company_objs=[]
# put any remainders
if len(company_objs) > 0:
    ndb.put_multi(company_objs)

这篇关于DeadlineExceededError：已超出响应HTTP请求的总体截止日期的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！