缓慢的Azure队列Webjob性能（本地开发） [英] Slow azure queue webjob performance (local dev)

问题描述

我的webapi必须执行一系列繁重的操作才能完成请求。为了最大程度地减少处理时间，我将视图计数器增量卸载到了Webjob。

我目前的操作方式是将userId和productId的消息加入队列在每个请求结束时进行队列存储。 webjob函数会在新的队列消息上触发，并且在解析消息后会向静态并发字典添加值（增加或添加新值）。

我不会递增并写入Azure表，因为我想使用另一个基于计时器的Webjob将并发字典中的值持久保存到表中，以避免过多的写入。

虽然没有消息丢失，并且所有值都已正确记录/添加到dict中，但是我发现性能很差。处理1000条消息需要5分钟以上的时间。 与将BatchSize设置为1相比，消息突发并且并行处理速度慢。我的webjob设置是：

  config.Queues.BatchSize = 32; 
 config.Queues.NewBatchThreshold = 100; 
 config.Queues.MaxPollingInterval = TimeSpan.FromSeconds（2）; 
 config.Queues.MaxDequeueCount = 3; 
  

由于字典冲突，性能似乎并没有受到影响，因为它们的处理速度很快。我的计时器显示，处理每个增量大约需要0到9毫秒，因此1000条消息最多需要9秒钟。 Webjob是滞后的还是闲置的，是设计使然的吗？ Webjob可以以什么最大速率处理队列？我还有其他设置可以调整以使Webjob运行更快吗？考虑到我可以有许多具有不同userIds / productIds的消息，还有其他方法可以执行此视图增量任务吗？

解决方案

< blockquote>

似乎在处理一组消息之后，在获取下一组消息之前，webjob的过程存在延迟

据我所知，WebJobs使用特定的轮询算法来轮询队列。找到一条消息后，SDK等待两秒钟，然后检查是否有另一条消息。如果找不到消息，它将等待约四秒钟，然后重试。有关详细信息，请检查轮询算法主题。 sdk-storage-queues-how＃a-idtriggera-如何在收到队列消息时触发功能 rel = nofollow noreferrer>本文。

此外，您可以尝试在Azure App Service Web应用程序中运行WebJob并将WebJobs扩展到多个实例。

My webapi must perform a set of heavy operations in order to fullfil a request. To minimize processing time I am offload the "view counter increment" to a webjob.

The way I am doing it currently is by enqueueing a message with userId and productId to azure queue storage at the end of each request. The webjob function triggers on new queue messages and after parsing the message it adds values (increments or adds new) to a static concurrent dictionary.

I am not incrementing and writing to azure table because I'd like to use another timer-based webjob to persist values from concurrent dictionary to the table to avoid excessive writes.

While no messages get lost and all values get recorded/added properly into dict, I am seeing very poor performance. It takes over 5 minutes to process 1000 messages. Messages come in bursts and are processed slower in parallel than when I set the BatchSize to 1. My webjob settings are:

config.Queues.BatchSize = 32;
config.Queues.NewBatchThreshold = 100;
config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(2);
config.Queues.MaxDequeueCount = 3;

The performance doesn't seem to suffer due to dictionary collisions as those seem to be processing fast. My timer shows that it takes between 0 and 9ms to process each increment, so 1000 messages should take 9 seconds at most. Is the webjob lagging/idling and is it by design? At what maximum rate can a webjob process the queue? Are there any other settings I can adjust to makes the webjob run faster? Are there other ways to go about this "view increment" task, considering that I can have many messages with different userIds/productIds?

解决方案

It seems that there's a delay in webjob's process after a set of messages is processed and before the next set is acquired

As far as I know, WebJobs poll the queue with specific polling algorithm. When a message is found, the SDK waits two seconds and then checks for another message; when no message is found it waits about four seconds before trying again. For detailed information, please check Polling algorithm topic in this article.

Besides, you could try to run WebJobs in a Azure App Service web app and scale WebJobs out to multiple instances.

这篇关于缓慢的Azure队列Webjob性能（本地开发）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！