如何在从Azure Storage Queue进行提取，推送和删除时提高性能 [英] How do I improve performance while doing pull, push and delete from Azure Storage Queue

问题描述

我正在使用Azure Storage Queue进行分布式应用程序以进行消息排队。队列将由多个客户端全天候使用，因此预计在使用时它将大量加载。业务案例是典型的，因为它从队列中提取消息，处理消息然后从队列中删除消息。该模块还向用户发回通知，指示进程已完成。功能/模块工作正常，因为它们符合逻辑要求。非常典型的队列场景。



现在，来看问题陈述。因为设想队列在大多数时间都会负载很重，所以我正在努力加快整个消息生命周期的处理速度。我可以更快地清除消息，为每个人，系统和用户提供更好的整体体验。



为了提高性能，我做了多个周期进行性能分析，然后改进已识别的HOT路径/功能。

这一切都归结为只有Azure Queue pull和delete是外面唯一两个最耗时的呼叫。我可以进一步改进pull，我通过批量拉动35条消息（这是我在编写此问题时可以立即从Azure队列中提取的最大消息数量）来做的。）这给了我一个帮助，就像在通过将处理时间缩短到很大的余地。这一切都很好。

我正在并行处理这些消息，以提高整体性能。



伪代码：

Hi,

I am working on a distributed application with Azure Storage Queue for message queuing. queue will be used by multiple clients across the clock and thus it is expected that it would be heavily loaded most on the time in usage. business case is typical as in it pulls message from queue, process the message then deletes the message from queue. this module also sends back a notification to user indicating process is complete. functions/modules work fine as in they meet the logical requirement. pretty typical queue scenario.

Now, coming to the problem statement. since it is envisaged that the queue would be heavily loaded most of the time, I am pushing towards to speed up processing of the overall message lifetime. the faster I can clear messages, the better overall experience it would be for everyone, system and users.

To improve on performance I did multiple cycles for performance profiling and then improving on the identified "HOT" path/function.
It all came down to a point where only the Azure Queue pull and delete are the only two most time consuming calls outside. I can further improve on pull, which i did by batch pulling 35 message at a time (which is the max message count i can pull from Azure queue at once at the time of writing this question.), this returned me a favor as in by reducing processing time to a big margin. all good till this as well.
i am processing these messages in parallel so as to improve on overall performance.

pseudo code:

//AzureQueue Class is encapsulating calls to Azure Storage Queue.
//assume nothing fancy inside, vanila calls to queue for pull/push/delete
var batchMessages = AzureQueue.Pull(35);            Parallel.ForEach(batchMessages, bMessage =>
            {
              //DoSomething does some background processing;
              try{DoSomething(bMessage);}
              catch()
              {
               //Log exception
              }
              AzureQueue.Remove(bMessage);
            });

现在进行此更改后，分析结果显示，最多90％的时间仅由Azure Message删除调用执行。由于最好在处理完成后立即删除消息，我会在DoSomething完成后将其删除。



我现在需要的是如何建议当Azure Queue Delete调用本身占用90％的时间时，可以进一步提高此功能的性能？有更好的更快的方式来执行删除/批量删除等？



这里提到的实现，我得到接近25个消息/秒的速度。目前，Azure队列删除调用正在扼杀应用程序性能。那么有没有希望进一步推动它。



如果您需要任何其他信息或有任何澄清，请告诉我。



欢迎输入/建议。



非常感谢。

With this change now, profiling results show that up-to 90% of time is only taken by the Azure Message delete calls. As it is good to delete message as soon as processing is done, i remove it just after "DoSomething" is finished.

what i need now is suggestions on how to further improve performance of this function when 90% of the time is being eaten up by the Azure Queue Delete call itself? is there a better faster way to perform delete/bulk delete etc?

with the implementation mentioned here, i get speed of close to 25 messages/sec. Right now Azure queue delete calls are choking application performance. so is there any hope to push it further.

Let me know if you need any additional information or any clarification in question.

Inputs/suggestions are welcome.

Many thanks.

推荐答案

hi
OmniSource



如果你在azure队列服务中获得性能提升，首先想到服务总线队列将是一个突出的回答这个？

但我错了，在某些情况下也是对的



请仔细阅读< a href =http://msdn.microsoft.com/en-us/library/azure/hh767287.aspx>链接



我的回答如果你可以减少消息大小本身，那么它可以做很多事情来唤醒性能！



因为你可以增加

hi OmniSource

If you come to the Performance increase in azure queues service , at first thought Service Bus Queue would be a prominent answer for this ?
but i was wrong and also right in some case

please go through this link

my answer would be if you can comprmoise in decreasing the Message Size itself then it can do alot to WAKE up the PERFORMANCE !

as you can increase the

引用：

吞吐量高达每秒2,000条消息

（bas基于1 KB消息的基准测试）

throughput up to 2,000 messages per second
(based on benchmark with 1 KB messages)

这篇关于如何在从Azure Storage Queue进行提取，推送和删除时提高性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！