使用Cloud Task，Cloud Functions和GAE快速创建云任务 [英] Rapidly creating cloud tasks using Cloud Task, Cloud Functions, and GAE

问题描述

我正在使用App Engine根据列表中的项目触发多个Cloud Task的创建:

I'm using App Engine to trigger the creation of multiple Cloud Tasks based on items in a list:

function_url="https://us-central1-myproject.cloudfunctions.net/some-cloud-function"

someTasks = [
{'id': 'task-1'},
{'id': 'task-2'},
{'id': 'task-3'},
...
{'id': 'task-1000'},
]

当前使用以下任务创建任务:

The tasks are currently created using:

Parallel(backend='threading', n_jobs=100)(
        delayed(create_document_task)(function_url=function_uri, data=task) for task in someTasks 
                    )

上面的代码并行创建任务，并指示任务队列将有效负载定向到该特定的云功能.

The above code creates the tasks in parallel and instructs the Task Queue to direct the payload to that specific cloud function.

并行执行此操作是否是快速创建任务的正确方法?

Is doing this in parallel the correct way to rapidly create tasks?

推荐答案

- I am posting this as an answer, due to the amount of text not fitting in a comment.

似乎上述(在评论中)方法:

It seems that the aforementioned (in the comments) method:

• Queue("someQueue").add_async(tasks)

确实是一种旧方法.此方法是在

is indeed an old method. This method is implemented within the Task Queue REST API (v1), in order to asynchronously add a task or list of tasks into a task queue.

不过，如此处所述，App Engine自2018年2月20日起，Task Queue REST API(v1)已被关闭.单独的产品Cloud Tasks提供了 REST API ，您可以使用它来添加第二代App Engine标准环境运行时，任何App Engine灵活环境运行时，甚至完全来自App Engine外部的任务

However as stated here, The App Engine Task Queue REST API (v1), was turned down as of February 20, 2018. The separate product Cloud Tasks provides a REST API that you can use to add tasks from a second generation App Engine standard environment run-time, any of the App Engine flexible environment run-times, or even from entirely outside of App Engine.

此API 不不包括"add_async()"功能.更具体地说，请此处和 App Engine SDK的用户拥有该选项，不是通过Cloud Tasks API可用的功能.

This API does not include the "add_async()" functionality. More specifically, here and here is affirmed that the feature of adding task to queues asynchronously, as the users of App Engine SDK have the option to do, is NOT an available feature via Cloud Tasks API.

尽管如此，当需要添加大量的云任务(例如数百万或数十亿)时，

Nonetheless, when a large number of Cloud Tasks, for example millions or billions, need to be added, a double-injection pattern can be useful.

要实现此方案，您需要创建一个新的注入器队列，该注入器队列的单个任务将包含用于添加要使用的原始队列的多个(100)任务的信息.在该注入器队列的接收端将是一个服务，该服务会将预期任务实际添加到您的原始队列中.尽管此服务中的任务添加是同步的，并且是1比1的，但它将为您的主应用程序提供一个异步接口以批量添加任务.通过这种方式，您可以克服主应用程序中同步添加1比1任务的局限性.

To implement this scenario, you'll need to create a new injector queue, whose single task would contain information to add multiple(100) tasks of the original queue that you're using. On the receiving end of this injector queue would be a service which does the actual addition of the intended tasks to your original queue. Although the addition of tasks in this service will be synchronous and 1-by-1, it will provide an asynchronous interface to your main application to bulk add tasks. In such a way you can overcome the limits of synchronous, 1-by-1 task addition in your main application.

请注意，建议将 500/50/5模式添加到队列中，以避免任何(队列/目标)重载.

Note that the 500/50/5 pattern of task addition to queue is a suggested method, in order to avoid any (queue/target) overloads.

由于我没有找到此实现的任何示例，因此我将在找到答案后立即对其进行编辑.

As I did not find any examples of this implementation, I will edit the answer as soon as I find one.

我希望这会有所帮助.

这篇关于使用Cloud Task，Cloud Functions和GAE快速创建云任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！