带有队列的长时间运行的 REST API [英] Long running REST API with queues

问题描述

我们正在实施 REST API，它将启动多个长时间运行的后端任务.我一直在阅读 RESTful Web 服务手册，建议返回 HTTP 202/Accepted 并带有指向正在处理的任务的 Content-Location 标头.(例如 http://www.example.org/orders/tasks/1234)，并让客户投票此 URI 用于更新长时间运行的任务.

We are implementing a REST API, which will kick off multiple long running backend tasks. I have been reading the RESTful Web Services Cookbook and the recommendation is to return HTTP 202 / Accepted with a Content-Location header pointing to the task being processed. (e.g. http://www.example.org/orders/tasks/1234), and have the client poll this URI for an update on the long running task.

这个想法是让 REST API 立即将消息发布到队列，后台工作角色从队列中获取消息并启动多个后端任务，也使用队列.我看到这种方法的问题是如何为任务分配一个唯一的 ID，然后让客户端通过向 Content-Location URI 发出 GET 请求来请求任务的状态.

The idea is to have the REST API immediately post a message to a queue, with a background worker role picking up the message from the queue and spinning up multiple backend tasks, also using queues. The problem I see with this approach is how to assign a unique ID to the task and subsequently let the client request a status of the task by issuing a GET to the Content-Location URI.

如果 REST API 立即发布到队列，那么它可以生成一个 GUID 并将其作为属性附加到添加到队列的消息上，但是获取请求的状态变得很尴尬.

If the REST API immediately posts to a queue, then it could generate a GUID and attach that as an attribute on the message being added to the queue, but fetching the status of the request becomes awkward.

另一种选择是让 REST API 立即向数据库添加一个条目(假设是一个带有新订单 ID 的订单)，具有初始状态，然后在队列中添加一条消息以启动后面地面任务，然后将更新该数据库记录.API 将在 Content-Location 标头的 URI 中返回此新订单 ID，供客户端在检查任务状态时使用.

Another option would be to have the REST API immediately add an entry to the database (let's say an order, with a new order id), with an initial status and then subsequently put a message on the queue to kick off the back ground tasks, which would then subsequently update that database record. The API would return this new order ID in the URI of the Content-Location header, for the client to use when checking the status of the task.

不知何故，首先添加数据库条目，然后将消息添加到队列中似乎是倒退，但仅将请求添加到队列中很难跟踪进度.

Somehow adding the database entry first, then adding the message to the queue seems backwards, but only adding the request to the queue makes it hard to track progress.

推荐的方法是什么?

非常感谢您的见解.

推荐答案

我假设您的系统如下所示.您有一个 REST 服务，它接收来自客户端的请求.它将请求转换为业务逻辑可以理解的命令.您将这些命令放入队列中.您有一个或多个工作人员可以处理这些命令并将其从队列中删除，并将结果发送到 REST 服务，该服务可以响应客户端.

I assume your system looks like the following. You have a REST service, which receives requests from the client. It converts the requests into commands which the business logic can understand. You put these commands into a queue. You have a single or multiple workers which can process and remove these commands from the queue and send the results to the REST service, which can respond to the client.

您的问题是您的长时间运行任务导致客户端连接超时，因此您无法发送响应.因此，您可以做的是在将命令放入队列并添加轮询链接后发送 202 接受，以便客户端能够轮询更改.您的任务有多个子任务，因此有进度，而不仅仅是待处理和已完成的状态更改.

Your problem that by your long running tasks the client connection timeouts, so you cannot send a response. So what you can do is sending a 202 accepted after you put the commands into the queue and add a polling link, so the client will be able to poll for the changes. Your tasks have multiple subtasks so there is progress, not just pending and complete status changes.

1. 如果您想坚持使用轮询，您应该创建一个新的 REST 资源，其中包含长时间运行的任务的实际状态和进度.这意味着您必须将此信息存储在数据库中，以便 REST 服务能够响应诸如 GET/tasks/23461/status 之类的请求.这意味着您的工作人员必须在完成子任务或整个任务后更新数据库.
2. 如果您的 REST 服务作为守护进程运行，那么您可以通过进度通知它，因此将任务状态存储在数据库中将不是工作人员的责任.这种 REST 服务也可以将信息存储在内存中.
3. 如果您决定使用 websockets 来通知客户端，那么您可以创建一个通知服务.通过 REST，您必须使用任务 ID 进行响应.之后你在 websocket 连接上发回这个任务 id，这样通知服务就会知道哪个 websocket 连接订阅了某个任务的事件.之后您就不需要 REST 服务了，只要客户端不关闭连接，您就可以通过 websocket 连接发送进度.
4. 您可以通过以下方式组合这些解决方案.您让 REST 服务创建任务资源，这样您就可以使用轮询链接访问进度.之后，您发回一个带有 202 的标识符，您通过 websockets 连接发回该标识符.所以你可以使用通知服务来通知客户端.随着进度，您的工作人员将通知 REST 服务，该服务将创建一个类似 GET/tasks/23461/status 的链接，并通过通知服务将该链接发送给客户端.之后，客户端可以使用该链接更新其状态.

1. If you want to stick with polling, you should create a new REST resource, which contains the actual state and the progress of the long running task. This means that you have to store this info in a database, so the REST service will be able to respond to requests like GET /tasks/23461/status. This means that your worker has to update the database when it is completed a subtask or the whole task.
2. If your REST service is running as a daemon, then you can notify it by progress, so storing the task status in the database won't be the responsibility of the worker. This kind of REST service can store the info in the memory as well.
3. If you decide to use websockets to notify the client, then you can create a notification service. By REST you have to respond with a task id. After that you send back this task id on the websocket connection, so the notification service will know which websocket connection subscribed to the events of a certain task. After that you won't need the REST service, you can send the progress through the websocket connection as long as the client does not close the connection.
4. You can combine these solutions the following way. You let your REST service to create a task resource, so you'll be able to access the progress by using a polling link. After that you send back an identifier with 202 which you send back through the websockets connection. So you can use a notification service to notify the client. By progress your worker will notify the REST service, which will create a link like GET /tasks/23461/status and send that link to the client through the notification service. After that the client can use the link to update its status.

如果您的 REST 服务作为守护进程运行，我认为最后一个是最好的解决方案.这是因为您可以将通知责任转移到专用的通知服务上，该服务可以使用 websockets、轮询、SSE，无论您想要什么.它可以在不杀死 REST 服务的情况下崩溃，因此 REST 服务将保持稳定和快速.如果您也使用 202 发回手动更新链接，那么客户端可以进行手动更新(假设是人为控制的客户端)，因此如果通知服务不可用，您将遇到类似优雅降级的情况.您不必维护通知服务，因为它对任务一无所知，它只会向客户端发送数据.您的工作人员无需了解有关如何发送通知和如何创建超链接的任何信息.维护客户端代码也会更容易，因为它几乎是一个纯粹的 REST 客户端.唯一的额外功能是订阅通知链接，不会经常更改.

I think the last one is the best solution if your REST service runs as a daemon. It is because you can move the notification responsibility to a dedicated notification service, which can use websockets, polling, SSE, whatever you want. It can collapse without killing the REST service, so the REST service will stay stable and fast. If you send back a manual update link too with the 202, then the client can do manual update (assuming a human controlled client), so you will have something like graceful degradation if the notification service is not available. You don't have to maintain the notification service because it won't know anything about the tasks, it will just send data to the clients. Your worker won't have to know anything about how to send notifications and how to create hyperlinks. It will be easier to maintain the client code too, since it will be almost a pure REST client. The only extra feature will be the subscription for the notification links, which does not change frequently.

这篇关于带有队列的长时间运行的 REST API的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！