Azure Functions 缩放和并发使用队列触发器和 functionAppScaleLimit 消耗计划 [英] Azure Functions scaling and concurrency using Queue triggers and functionAppScaleLimit on the Consumption Plan

问题描述

我在 Linux 消费计划上有一个 Azure Function 应用，它有两个队列触发器.两个队列触发器都将 batchSize 参数设置为 1 因为它们每个都可以使用大约 500 MB 的内存而且我不想超过 1.5GB 的内存限制，所以他们一次只能接收一条消息.

如果我想允许这两个队列触发器同时运行，但不希望它们超出此范围，请将 functionAppScaleLimit 设置为 2 足以做到了吗?

添加新示例，感谢@Hury Shen 为这些示例提供框架

更多详情请查看下方@Hury Shen 的回答.我已经测试了三个队列触发场景.全部使用以下图例:

QueueTrigger 没有 functionAppScaleLimit

QueueTrigger 与 functionAppScaleLimit 设置为 2
QueueTrigger 与 functionAppScaleLimit 设置为 1

现在，我想我会坚持最后一个例子，但在未来我想我可以安全地将我的 functionAppScaleLimit 设置为 2 或 3 如果我升级到高级计划.我还将测试两个队列触发器，它们使用 functionAppScaleLimit 的 2 侦听不同的存储队列，但我怀疑对我来说最安全的做法是创建单独的 Azure该场景中每个队列触发器的函数应用程序.

编辑 2:在一个函数应用中添加两个队列触发器的示例

以下是在一个 Azure 函数中使用两个队列触发器时的结果，这些触发器正在侦听两个不同的存储队列.这是两个队列触发器的图例:

两个队列触发器同时运行，functionAppScaleLimit 设置为 2
两个队列触发器同时运行，functionAppScaleLimit 设置为 1

在两个队列触发器同时运行且 functionAppScaleLimit 设置为 2 的示例中，看起来比例限制不起作用.微软的人可以解释一下吗?官方文档中没有警告(

我在上面截图的右边使用了 4 个红框，我将四个日志中的每一个命名为s1"，s2"，s3"、s4"(步骤 1-4).并总结excel中的日志供大家参考:

我用s2"制作细胞到s4"为黄色，因为这段时间是指函数任务的执行时间.

根据excel的截图，我们可以推断出以下几点:

1.实例的最大数量只能扩展到2，因为我们可以发现它在excel表格的每一行中不超过两个黄色单元格.因此，正如您在问题中提到的那样，该功能无法扩展到超过 2 个实例.

2.你想让这两个队列触发器同时运行，可以实现.但是实例会通过消费机制进行横向扩展. 简单来说，当一个函数实例被一条消息触发，还没有完成任务，现在又进来了一条消息，就不能确保使用另一个实例.第二条消息可能正在等待第一个实例.我们无法控制是否启用另一个实例.

================================更新===============================

由于我对您的描述不是很清楚，我不确定您要收听多少个存储队列，以及您在自己身边创建了多少个函数应用程序和 QueueTrigger 函数.我将我的测试结果总结如下，供大家参考:

1.关于您在高级计划中描述的最大突发的行为是否与此不同?我认为如果我们选择高级计划，实例也会使用相同的消费计划机制进行横向扩展.

2.如果你有两个存储队列需要监听，当然我们应该创建两个QueueTrigger函数来监听每个存储队列.

3.如果你只需要监听一个存储队列，我用三种情况进行测试(我在三种情况下都将最大规模实例设置为2):

A) 在one函数应用中创建oneQueueTrigger函数来监听one存储队列.这是我在原始答案中测试的，excel表格显示实例将通过消费计划机制向外扩展，我们无法控制.

B) 在一个函数应用中创建两个QueueTrigger函数来监听同一个存储队列.结果与案例A几乎相同，我们无法控制使用多少个实例来处理消息.

C) 创建两个函数应用并在每个函数应用中创建一个QueueTrigger函数来收听相同 存储队列.结果也和案例 A 和 B 类似，不同的是最大实例可以扩展到 4，因为我创建了两个函数应用程序(它们都可以扩展到 2 个实例).

总之，我认为这三种情况都是相似的.虽然我们选择案例 3，但创建两个函数应用程序，每个应用程序中都有一个 QueueTrigger 函数.我们也不能确保第二条消息立即处理，它仍然可能被处理到第一个实例并等待第一个实例完成处理第一条消息.

所以这篇文章中您当前问题的答案 是否将 functionAppScaleLimit 设置为 2 足以实现这一目标? 是:如果您希望两个实例都是启用并发运行，我们无法确定.如果你只想要两个实例来处理消息，答案是肯定的.

I have an Azure Function app on the Linux Consumption Plan that has two queue triggers. Both queue triggers have the batchSize parameter set to 1 because they can both use about 500 MB of memory each and I don't want to exceed the 1.5GB memory limit, so they should only be allowed to pick up one message at a time.

If I want to allow both of these queue triggers to run concurrently, but don't want them to scale beyond that, is setting the functionAppScaleLimit to 2 enough to achieve that?

Edit: added new examples, thank you @Hury Shen for providing the framework for these examples

Please see @Hury Shen's answer below for more details. I've tested three queue trigger scenarios. All use the following legend:

QueueTrigger with no functionAppScaleLimit

QueueTrigger with functionAppScaleLimit set to 2
QueueTrigger with functionAppScaleLimit set to 1

For now, I think I'm going to stick with the last example, but in the future I think I can safely set my functionAppScaleLimit to 2 or 3 if I upgrade to the premium plan. I also am going to test two queue triggers that listen to different storage queues with a functionAppScaleLimit of 2, but I suspect the safest thing for me to do is to create separate Azure Function apps for each queue trigger in that scenario.

Edit 2: add examples for two queue triggers within one function app

Here are the results when using two queue triggers within one Azure Function that are listening on two different storage queues. This is the legend for both queue triggers:

Both queue triggers running concurrently with functionAppScaleLimit set to 2
Both queue triggers running concurrently with functionAppScaleLimit set to 1

In the example where two queue triggers are running concurrently with functionAppScaleLimit set to 2 it looks like the scale limit is not working. Can someone from Microsoft please explain? There is no warning in the official documentation (https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#limit-scale-out) that this setting is in preview mode, yet we can clearly see that the Azure Function is scaling out to 4 instances when the limit is set to 2. In the following example, it looks like the limit is being respected, but the functionality is not what I want and we still see the waiting that is present in @Hury Shen's answer.

Conclusion
To limit concurrency and control scaling in Azure Functions with queue triggers, you must limit your Azure Function to use one queue trigger per function app and use the batchSize and functionAppScaleLimit settings. You will encounter race conditions and waiting that may lead to timeouts if you use more than one queue trigger.

解决方案

Yes, you just need to set functionAppScaleLimit to 2. But there are some mechanisms about consumption plan you need to know. I test it in my side with batchSize as 1 and set functionAppScaleLimit to 2(I set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT as 2 in "Application settings" of function app instead of set functionAppScaleLimit, they are same). And I test with the code below:

import logging
import azure.functions as func
import time

def main(msg: func.QueueMessage) -> None:
    logging.info('=========sleep start')
    time.sleep(30)
    logging.info('=========sleep end')
    logging.info('Python queue trigger function processed a queue item: %s',
                 msg.get_body().decode('utf-8'))

Then I add message to the queue, I sent 10 messages: 111, 222, 333, 444, 555, 666, 777, 888, 999, 000, I sent them one by one. The function was triggered success and after a few minutes, we can see the logs in "Monitor". Click one of the log in "Monitor", we can see the logs show as:

I use 4 red boxes on the right of the screenshot above, I named each of the four logs as "s1", "s2", "s3", "s4"(step 1-4). And summarize the logs in excel for your reference:

I make cells from "s2" to "s4" as yellow because this period of time refer to the execution time of the function task.

According the screenshot of excel, we can infer the following points:

1. The maximum number of instances can only be extended to 2 because we can find it doesn't exist more than two yellow cells in each line of the excel table. So the function can not scale beyond 2 instances as you mentioned in your question.

2. You want to allow both of these queue triggers to run concurrently, it can be implemented. But the instances will be scale out by mechanism of consumption. In simple terms, when one function instance be triggered by one message and hasn't completed the task, and now another message come in, it can not ensure another instance be used. The second message might be waiting on the first instance. We can not control whether another instance is enabled or not.

===============================Update==============================

As I'm not so clear about your description, I'm not sure how many storage queues you want to listen to and how many function apps and QueueTrigger functions you created in your side. I summarize my test result as below for your reference:

1. For your question about would the Maximum Burst you described on the premium plan behave differently than this ? I think if we choose premium plan, the instances will also be scale out with same mechanism of consumption plan.

2. If you have two storage queues need to be listen to, of course we should create two QueueTrigger functions to listen to each storage queue.

3. If you just have one storage queue need to be listen to, I test with three cases(I set max scale instances as 2 in all of three cases):

A) Create one QueueTrigger function in one function app to listen to one storage queue. This is what I test in my original answer, the excel table shows us the instances will scale out by mechanism of consumption plan and we can not control it.

B) Create two QueueTrigger functions in one function app to listen to same storage queue. The result is almost same with case A, we can not control how many instances be used to deal with the messages.

C) Create two function apps and create a QueueTrigger function in each of function app to listen to same storage queue. The result also similar to case A and B, the difference is the max instances can be scaled to 4 because I created two function apps(both of them can scale to 2 instances).

So in a word, I think all of the three cases are similar. Although we choose case 3, create two function apps with one QueueTrigger function in each of them. We also can not make sure the second message be deal with immediately, it still may be processed to first instance and wait for frist instance complete deal with the first message.

So the answer for your current question in this post is setting the functionAppScaleLimit to 2 enough to achieve that? is: If you want both of instances be enabled to run concurrently, we can't make sure of it. If you just want two instances to deal with the messages, the answer is yes.

这篇关于Azure Functions 缩放和并发使用队列触发器和 functionAppScaleLimit 消耗计划的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！