从GAE中的任务获取数据以形成任务队列循环的最佳方式是什么？ [英] What's the best way to get data back from a Task in GAE to form a loop of Task Queues?

问题描述

在导致DeadlineExceededError的任务队列中，我有一个长时间运行的进程，因为任务可能需要超过10分钟。正如我在这个问题，长时间运行的进程有一个for循环，它依次计算用于创建kml文件的大型字典中的新值。这个任务目前看起来像这样：

pre $ class $ longprocess_handler（webapp2.RequestHandler）
def post（self）：
＃这是一个任务递归地使用字典中的数据到
＃每隔几分钟生产一次kml文件
在范围内（0，时间）j：
＃处理字典中的数据在此顺序循环
＃发送消息到客户端以请求数据。

我想将流程分成几个较小的任务，例如：

  class longprocess_handler（webapp2.RequestHandler）：
 def post（self）：
 for j in range（0，Time）：
＃将字典发送到较小的任务
 CallSmallerTask_handler（字典）
＃从任务接收字典。 （我该怎么做？）
＃重复以循环使用新字典来调用下一个任务。 
  

有没有办法从任务中取回数据，以便我可以创建一个小任务循环使用前一个任务的结果按顺序创建？我是否需要将前一个任务中的字典存储在数据存储中，然后检索它们以创建下一个任务？ （我希望避免这种情况，因为字典非常大，而且我认为这可能很困难）。 必须将大型字典存储到数据存储中。只有在一个任务的输出相对较小（而不是您的情况）的情况下，可以将输出作为参数传递给任务的处理程序。

一种可能的优化是使用ndb或在数据存储区顶部实现自己的缓存层，因此很大一部分读取调用将从缓存中完成，并且永远不会到达数据存储区。请记住，应该对缓存和数据存储进行存储，因为缓存不是100％可靠的。

I have a long-running process in a task queue that is causing a DeadlineExceededError because the task can take longer than 10 minutes. As I described in this question, the long-running process has a for loop that sequentially calculates new values in large dictionaries that are used to create kml files. The task currently looks like this:

class longprocess_handler(webapp2.RequestHandler):
def post(self):
#This is currently one task that recursively uses data in dictionaries to 
#produce kml files every few minutes
    for j in range(0, Time):
    # Processes data from dictionaries sequentially in this for loop
    # Sends message to client to request the data.

I would like to make the process into several smaller tasks such as:

class longprocess_handler(webapp2.RequestHandler):
def post(self):
    for j in range(0, Time):
        # Send dictionaries to smaller task 
        CallSmallerTask_handler(dictionaries)
        # Receive dictionaries back from task.  (How do I do this?)
        # Repeat for loop with new dictionaries to call next task.

Is there a way to get data back from a Task so that I can create a loop of smaller tasks that are created sequentially using the result of the previous task? Would I need to store the dictionaries from the previous task in the Datastore and then retrieve them to create the next task? (I am hoping to avoid this because the dictionaries are very large, and I think it may be difficult).

解决方案

You will have to store the large dictionaries into datastore. Only in the cases where the output of one task is relatively small (not your case), it is possible to pass the output as parameter(s) to the Task's handler.

One possible optimization is to use ndb or implement your own caching layer on top of the datastore so a good percentage of the read calls will be made from the cache and never reach the datastore. Just keep in mind that storing should be made to the cache and to the datastore since the cache is not 100% reliable.

这篇关于从GAE中的任务获取数据以形成任务队列循环的最佳方式是什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！