确保为特定队列扩展的实例数不超过N个 [英] ensuring no more than N instances are scaled out for a specific queue

问题描述

我的功能是将有效载荷发送到不同的sftp服务器。这些服务器在可接受的连接数量上受到限制。

我需要一种解决方案来限制与这些服务器的连接。

该功能由存储队列触发，设计的第一稿为：

然后我了解到每个功能只能有1个触发器，这使我陷入了另一个聚集队列：

我可以设置 batchSize / newBatchThreshold ，但是我我不确定这是否行得通，因为始发队列不会知道何时将邮件推送到聚合队列。

1. 我需要功能对于来自队列X的所有消息，请不要扩展到超过N个实例，因为sftp服务器X不能接受超过N个连接。


2. 此外，我需要扩展函数对于来自队列Y的所有消息，最多只能有M个实例，因为sftp服务器Y最多只能接受M个连接。

在上述情况下，实例总数为M + N。

我们如何调整设计以满足这些要求？

解决方案

没有100％为此，请在此处跟踪该问题。 / p>

最好的方法是将 WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT 设置为 1 聚合队列触发的功能应用程序的应用程序设置。然后，您应该只获得一个功能应用程序的并发实例，因此 batchSize 设置实际上对于限制速率很有用。

在这种情况下，您不需要限制队列处理器X / Y / Z，让消息流向聚合。

现在，我没有无法理解仅来自队列X的消息是否触及SFPT X，还是多对多。如果是一对一的，则有必要摆脱聚合队列，拥有三个功能并分别限制每个队列的并发性。

限制设置如我上面的建议。

如果仍然不能满足您的要求，则可以切换到其他消息服务。例如，将一种类型的所有消息发送到服务总线的单独会话或事件中心的单个分区中，这自然会限制代理级别的并发性。

My function is sending a payload to different sftp servers. Those servers are limited in how many connections they can accept.

I need a solution to throttle our connections to those servers.

The function is triggered by storage queues, and the first draft of the design is:

I then learned that you can only have 1 trigger per function, which led me to sandwhich another aggregating queue:

I can set the batchSize/newBatchThreshold on the originating queues, but I'm not certain this will work because the originating queues will not be aware of when to push messages to the aggregate queue.

1. I need the function to not scale out to more than N instances for all messages from queue X, since the sftp server X will not accept more than N connections.
2. Additionally, I need the function to scale out to no more than M instances for all messages from queue Y, since the sftp server Y will not accept more than M connections.

The total instances would be M + N for the above scenario.

How do we adjust our design in order to fit these requirements?

解决方案

There's no 100% bullet-proof solution to this, the issue is tracked here.

The best way could be setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 in application settings of the Function App which is triggered by the aggregate queue. Then, you should only get one concurrent instance of the Function App, so the batchSize setting will actually be useful for rate limiting.

You don't need to limit queue processors X/Y/Z in this case, let the messages flow to the aggregate.

Now, I didn't understand if only messages from queue X touch SFPT X, or it's many-to-many. If it's one-on-one, it makes sense to get rid of the aggregate queue, have three Functions and limit the concurrency for each of the queues separately.

Anyway, the limit settings are as I suggested above.

In case this still doesn't satisfy your requirements, you may switch to another messaging service. For instance, send all messages of one type into a separate session of Service Bus or a single partition of Event Hub, which will naturally limit the concurrency on the broker level.

这篇关于确保为特定队列扩展的实例数不超过N个的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！