如何从Google App Engine中的延期任务中返回数据 [英] How do I return data from a deferred task in Google App Engine

问题描述

原始问题

我有一个我正在尝试升级的Web应用程序的工作版本，而且我遇到了一个问题：在单个HTTP请求期间花费很长时间才能完成的任务。应用程序通过HTTP Post操作从JavaScript前端获取JSON列表，并返回该列表的排序/切片版本。随着输入列表变长，排序操作需要更长的时间来执行（显然），所以在适当长的输入列表中，我遇到了60秒的HTTP请求超时，并且应用程序失败。

我想开始使用延迟库来执行排序任务，但我不清楚在执行该任务后如何存储/检索数据。这是我目前的代码：

  class getLineups（webapp2.RequestHandler）：
 def post（self）：
 jsonstring = self.request.body 
 inputData = json.loads（jsonstring）
 playerList = inputData [pList] 
 positions = [QB，RB，WR ，TE，DST] 
 
 playersPos = sortByPos（playerList，positions）
 rosters，playerUse = getNFLRosters（playersPos，positions）
 try：
 ＃这一步计算起来很昂贵，在大型玩家名单上会失败。 
 lineups = makeLineups（rosters，playerUse，50000）
 
 self.response.headers [Content-Type] =application / json
 self.response.out。写（json.dumps（阵容））
，除了：
 logging.error（在播放列表长度达到60秒超时：，len（playerList））
 self.response.headers [Content-Type] =text / plain
 self.response.set_status（504）
 
 app = webapp2.WSGIApplication（[
（'/ lineup'， getLineups），
]，debug = True）
  

理想情况下，尝试/除了阻止调用延期任务库：

  deferred.defer（makeLineups，rosters，playerUse，50000）
  

但我不清楚如何从该操作中获得结果。我想我将不得不将它存储在数据存储中，然后检索它，但是我的JavaScript前端如何知道操作何时完成？我已经阅读了Google网站上的文档，但我仍然对如何完成此任务感到朦胧。

我如何解决这个问题

在接受的答案中使用基本大纲，以下是我解决此问题的方法：

$ $ $ $ $ $ c $ def defResult（ result_key）：
result = result_key.get（）

playersPos = sortByPos（result.playerList，result.positions）
rosters，playerUse = getNFLRosters（playersPos，result.positions）

lineups = makeLineups（rosters，playerUse，50000）
storeResult（result_key，lineups）

$ ndb.transactional
def storeResult（result_key，lineups ）：
result = result_key.get（）
result.lineups = lineups
result.solveComplete = True
result.put（）

class结果（ndb.Model）：
playerList = ndb.JsonProperty（）
positions = ndb.JsonProperty（）
solveComplete = ndb.BooleanProperty（）
$ b $ class getLineups（ webapp2.RequestHandler）：
def post（self）：
jsonstring = self.request.body
inputData = json.loads（jsonstring）

deferredResult =结果（
playerList = inputData [pList]，
位置= [QB，RB，WR，TE，DST]，
solveComplete = False
）

deferredResult_key = deferredResult.put （）

deferred.defer（solveResult，deferredResult_key）

self.response.headers [Content-Type] =text / plain
self。 response.out.write（deferredResult_key.urlsafe（））
$ b $ class queryResults（webapp2.RequestHandler）：
def post（self）：
safe_result_key = self.request.body
result_key = ndb.Key（urlsafe = safe_result_key）

result = result_key.get（）
self.response.headers [Content-Type] =application / json

如果result.solveComplete：
self.response.out.write（json.dumps（result.lineups））
else：
self.response.out。写（json.dumps（[]））

然后，JavaScript前端轮询queryLineups URL一段固定的时间，并在时间限制到期或停止轮询时停止轮询。我希望这对任何试图解决类似问题的人都有帮助。如果事情变得松散，我还有更多的工作要做，以便让它顺利失败，但是这样做并且需要改进。

解决方案

我对GAE并不熟悉，但这是一个相当普遍的问题，所以我可以给你一些建议。

你的一般想法是正确的，所以我是只是要扩大它。工作流程可能如下所示：

1. 您可以获得创建阵容的请求。您在数据存储中为它创建一个新实体。它应该包含一个ID（您将需要它以稍后检索结果）和状态（PENDING | DONE | FAILED）。您也可以保存请求中的数据（如果这对您有用）。


2. 您推迟计算并立即返回响应。响应将包含任务的ID。计算完成后，它会将任务的结果保存到数据存储中并更新任务的状态。该结果将包含任务ID，以便我们可以轻松找到它。


3. 一旦前端接收到ID，它就开始轮询结果。使用 setTimeout 或 setInterval ，您将带有任务ID的请求发送到服务器（这是一个单独的端点）。服务器检查任务的状态，如果完成则返回结果（如果失败，则返回错误）。


4. 前端获取数据并停止轮询。

Original Question

I have a working version of my web application that I am trying to upgrade at the moment, and I'm running into the issue of having a task which takes too long to complete in during a single HTTP request. The application takes a JSON list from a JavaScript front end by an HTTP Post operation, and returns a sorted/sliced version of that list. As the input list gets longer, the sorting operation takes a much longer time to perform (obviously), so on suitably long input lists, I hit the 60 second HTTP request timeout, and the application fails.

I would like to start using the deferred library to perform the sort task, but I'm not clear on how to store/retrieve the data after I perform that task. Here is my current code:

class getLineups(webapp2.RequestHandler):
  def post(self):
    jsonstring = self.request.body
    inputData = json.loads(jsonstring)
    playerList = inputData["pList"]
    positions = ["QB","RB","WR","TE","DST"]

    playersPos = sortByPos(playerList,positions)
    rosters, playerUse = getNFLRosters(playersPos, positions)
    try:
      # This step is computationally expensive, it will fail on large player lists.
      lineups = makeLineups(rosters,playerUse,50000)

      self.response.headers["Content-Type"] = "application/json"
      self.response.out.write(json.dumps(lineups))
    except:
      logging.error("60 second timeout reached on player list of length:", len(playerList))
      self.response.headers["Content-Type"] = "text/plain"
      self.response.set_status(504)

app = webapp2.WSGIApplication([
  ('/lineup',getLineups),
], debug = True)

Ideally I would like to replace the entire try/except block with a call to the deferred task library:

deferred.defer(makeLineups,rosters,playerUse,50000)

But I'm unclear on how I would get the result back from that operation. I'm thinking I would have to store it in the Datastore, and then retrieve it, but how would my JavaScript front end know when the operation is complete? I've read the documentation on Google's site, but I'm still hazy on how to accomplish this task.

How I Solved It

Using the basic outline in the accepted answer, here's how I solved this problem:

def solveResult(result_key):
  result = result_key.get()

  playersPos = sortByPos(result.playerList, result.positions)
  rosters, playerUse = getNFLRosters(playersPos,result.positions)

  lineups = makeLineups(rosters,playerUse,50000)
  storeResult(result_key,lineups)

@ndb.transactional
def storeResult(result_key,lineups):
  result = result_key.get()
  result.lineups = lineups
  result.solveComplete = True
  result.put()

class Result(ndb.Model):
  playerList = ndb.JsonProperty()
  positions = ndb.JsonProperty()
  solveComplete = ndb.BooleanProperty()

class getLineups(webapp2.RequestHandler):
  def post(self):
    jsonstring = self.request.body
    inputData = json.loads(jsonstring)

    deferredResult = Result(
      playerList = inputData["pList"],
      positions = ["QB","RB","WR","TE","DST"],
      solveComplete = False
    )

    deferredResult_key = deferredResult.put()

    deferred.defer(solveResult,deferredResult_key)

    self.response.headers["Content-Type"] = "text/plain"
    self.response.out.write(deferredResult_key.urlsafe())

class queryResults(webapp2.RequestHandler):
  def post(self):
    safe_result_key = self.request.body
    result_key = ndb.Key(urlsafe=safe_result_key)

    result = result_key.get()
    self.response.headers["Content-Type"] = "application/json"

    if result.solveComplete:
      self.response.out.write(json.dumps(result.lineups))
    else:
      self.response.out.write(json.dumps([]))

The Javascript frontend then polls queryLineups URL for a fixed amount of time and stops polling if either the time limit expires, or it receives data back. I hope this is helpful for anyone else attempting to solve a similar problem. I have a bit more work to do to make it fail gracefully if things get squirrelly, but this works and just needs refinement.

解决方案

I'm not familiar with GAE, but this is a fairly generic question, so I can give you some advice.

Your general idea is correct, so I'm just going to expand on it. The workflow could look like this:

1. You get the request to create the lineups. You create a new entity in the datastore for it. It should contain an ID (you'll need it to retrieve the result later) and a status (PENDING|DONE|FAILED). You can also save the data from the request, if that's useful to you.
2. You defer the computation and return a response right away. The response will contain the ID of the task. When the computation is done, it will save the result of the task in the Datastore and update the status of the task. That result will contain the task ID, so that we can easily find it.
3. Once the frontend receives the ID, it starts polling for the result. Using setTimeout or setInterval you send requests with the task ID to the server (this is a separate endpoint). The server checks the status of the task, and returns the result if it's done (error if failed).
4. The frontend gets the data and stops polling.

这篇关于如何从Google App Engine中的延期任务中返回数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！