限制Azure Functions队列上的并发作业数 [英] Limiting the number of concurrent jobs on Azure Functions queue

问题描述

我在Azure中有一个功能应用程序，当将项目放入队列时会触发该应用程序。看起来像这样（大大简化了）：

I have a Function app in Azure that is triggered when an item is put on a queue. It looks something like this (greatly simplified):

public static async Task Run(string myQueueItem, TraceWriter log)
{
    using (var client = new HttpClient())
    {
        client.BaseAddress = new Uri(Config.APIUri);
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

        StringContent httpContent = new StringContent(myQueueItem, Encoding.UTF8, "application/json");
        HttpResponseMessage response = await client.PostAsync("/api/devices/data", httpContent);
        response.EnsureSuccessStatusCode();

        string json = await response.Content.ReadAsStringAsync();
        ApiResponse apiResponse = JsonConvert.DeserializeObject<ApiResponse>(json);

        log.Info($"Activity data successfully sent to platform in {apiResponse.elapsed}ms.  Tracking number: {apiResponse.tracking}");
    }
}

这一切都很好，并且运行良好。每次将项目放入队列时，我们都会将数据发送到我们这边的某个API并记录响应。不错。

This all works great and runs pretty well. Every time an item is put on the queue, we send the data to some API on our side and log the response. Cool.

问题发生在生成队列消息的事物急剧增加并且很多项目被同时放入队列时。这往往在一分钟内发生约1,000到1,500个项目。错误日志将具有以下内容：

The problem happens when there's a big spike in "the thing that generates queue messages" and a lot of items are put on the queue at once. This tends to happen around 1,000 - 1,500 items in a minute. The error log will have something like this:

2017-02-14T01：45：31.692 mscorlib：执行函数时异常：
Functions.SendToLimeade。 f-SendToLimeade __- 1078179529：发送请求时发生错误
。系统：无法连接到
远程服务器。系统：通常只允许每个套接字地址
（协议/网络地址/端口）使用一次
123.123.123.123:443。

2017-02-14T01:45:31.692 mscorlib: Exception while executing function: Functions.SendToLimeade. f-SendToLimeade__-1078179529: An error occurred while sending the request. System: Unable to connect to the remote server. System: Only one usage of each socket address (protocol/network address/port) is normally permitted 123.123.123.123:443.

首先，我认为这是Azure Function应用程序用尽本地套接字的问题，如在此说明。但是，然后我注意到了IP地址。 IP地址123.123.123.123（此示例中当然有所更改）是我们的IP地址，即HttpClient发布到的IP地址。所以，现在我想知道是否是我们个服务器用尽了套接字来处理这些请求。

At first, I thought this was an issue with the Azure Function app running out of local sockets, as illustrated here. However, then I noticed the IP address. The IP address 123.123.123.123 (of course changed for this example) is our IP address, the one that the HttpClient is posting to. So, now I'm wondering if it is our servers running out of sockets to handle these requests.

无论哪种方式，我们都可以扩展问题在这里继续。我正在尝试找出解决问题的最佳方法。

Either way, we have a scaling issue going on here. I'm trying to figure out the best way to solve it.

一些想法：

1. 如果是本地套接字限制，则给出了使用 Req增加本地端口范围的示例。 ServicePoint.BindIPEndPointDelegate 。这似乎很有希望，但是当您真正需要扩展时该怎么办？我希望这个问题不会在2年后再次出现。


2. 如果是一个远程限制，看来我可以控制功能运行时一次处理多少条消息。这里有一篇有趣的文章，说您可以将 serviceBus.maxConcurrentCalls 设置为1，一次只处理一条消息。也许我可以将其设置为相对较低的数字。现在，到某个时候，我们的队列将比处理它们的速度快得多，但是到那时，答案是在我们的末端添加了更多服务器。


3. 多个Azure Functions应用程序？如果我有多个Azure Functions应用程序，并且它们都在同一队列中触发，该怎么办？ Azure是否足够聪明，可以在Function应用程序之间分配工作，我是否可以拥有大量机器来处理我的队列，可以根据需要将其放大或缩小？


4. 我还遇到保持活力。在我看来，如果我能以某种方式保持我的套接字在队列消息泛滥时保持打开状态，那可能会很有帮助。这可能吗？有关我将如何执行此操作的任何提示？

1. If it's a local socket limitation, the article above has an example of increasing the local port range using Req.ServicePoint.BindIPEndPointDelegate. This seems promising, but what do you do when you truly need to scale? I don't want this problem coming back in 2 years.
2. If it's a remote limitation, it looks like I can control how many messages the Functions runtime will process at once. There's an interesting article here that says you can set serviceBus.maxConcurrentCalls to 1 and only a single message will be processed at once. Maybe I could set this to a relatively low number. Now, at some point our queue will be filling up faster than we can process them, but at that point the answer is adding more servers on our end.
3. Multiple Azure Functions apps? What happens if I have more than one Azure Functions app and they all trigger on the same queue? Is Azure smart enough to divvy up the work among the Function apps and I could have an army of machines processing my queue, which could be scaled up or down as needed?
4. I've also come across keep-alives. It seems to me if I could somehow keep my socket open as queue messages were flooding in, it could perhaps help greatly. Is this possible, and any tips on how I'd go about doing this?

对推荐（可扩展！）设计的任何见解

Any insight on a recommended (scalable!) design for this sort of system would be greatly appreciated!

推荐答案

我想我已经找到了解决方案。在过去的 3个小时 6个小时内，我一直在进行这些更改，并且套接字错误为零。在每30分钟左右大批量获取这些错误之前。

I think I've figured out a solution for this. I've been running these changes for the past 3 hours 6 hours, and I've had zero socket errors. Before I would get these errors in large batches every 30 minutes or so.

首先，我添加了一个新类来管理HttpClient。

First, I added a new class to manage the HttpClient.

public static class Connection
{
    public static HttpClient Client { get; private set; }

    static Connection()
    {
        Client = new HttpClient();

        Client.BaseAddress = new Uri(Config.APIUri);
        Client.DefaultRequestHeaders.Add("Connection", "Keep-Alive");
        Client.DefaultRequestHeaders.Add("Keep-Alive", "timeout=600");
        Client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    }
}

现在，我们有一个<$ c的静态实例$ c> HttpClient ，该函数用于每次调用该函数。根据我的研究，强烈建议将HttpClient实例保持尽可能长的时间，一切都是线程安全的，并且HttpClient会将请求排队并优化对同一主机的请求。注意，我还设置了 Keep-Alive 标头（我认为这是默认值，但我想我会隐含）。

Now, we have a static instance of HttpClient that we use for every call to the function. From my research, keeping HttpClient instances around for as long as possible is highly recommended, everything is thread safe, and HttpClient will queue up requests and optimize requests to the same host. Notice I also set the Keep-Alive headers (I think this is the default, but I figured I'll be implicit).

在我的函数中，我只是抓住了静态HttpClient实例，例如：

In my function, I just grab the static HttpClient instance like:

var client = Connection.Client;
StringContent httpContent = new StringContent(myQueueItem, Encoding.UTF8, "application/json");
HttpResponseMessage response = await client.PostAsync("/api/devices/data", httpContent);
response.EnsureSuccessStatusCode();

我真的没有对套接字级别发生的情况进行任何深入的分析（我我将不得不问问我们的IT人员是否能够在负载均衡器上看到此流量），但我希望它可以使我们的服务器保持一个套接字开放，并在处理队列项目时进行大量HTTP调用。无论如何，无论做什么，似乎都行得通。也许有人对如何改进有所想法。

I haven't really done any in-depth analysis of what's happening at the socket level (I'll have to ask our IT guys if they're able to see this traffic on the load balancer), but I'm hoping it just keeps a single socket open to our server and makes a bunch of HTTP calls as the queue items are processed. Anyway, whatever it's doing seems to be working. Maybe someone has some thoughts on how to improve.

这篇关于限制Azure Functions队列上的并发作业数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！