Azure存储队列-处理时间长 [英] Azure Storage Queue - long time to process

问题描述

我需要生成大量报告，并且一个报告可能需要大约5分钟才能生成，大量数据以及许多不同的来源.

I need to generate quite a number of reports and a report can take about 5 minutes to be generated, large amount of data, many different sources.

客户端将消息发布到Azure存储队列.有一个工作人员角色来处理消息并生成报告.

The client will post messages to an Azure Storage Queue. There is a worker roles that processes the messages and generates the reports.

如果我想扩大规模，可以说我有10个工作角色，这些角色将处理队列中的消息并生成报告.然后，我将消息添加到队列中，如下所示:

If I want to scale this up let's say I end up with 10 worker roles that will process the messages from the queue and generate the reports. Then I will add messages into the queue like this:

• 消息1:处理1-5中的报告
• 消息2:处理6-11中的报告 ........
• 消息10:处理50-55之间的报告(范围可能不准确)

• message 1: process reports from 1 - 5
• message 2: process reports from 6 - 11 ........
• message 10: process reports from 50 - 55 (might not be accurate the range)

如果我的辅助角色1接收了第一条消息并对其进行了锁定，但是该过程将花费5分钟，则该锁定将过期并且该消息将在队列中再次可见，因此辅助角色2将对其进行处理，然后开始处理它……依此类推

If my worker role 1 will take the first message and put a lock on it but the process will take 5 minutes, the lock will expire and the message will be visible again in the queue so the worker role 2 will take it and start processing it ... and so forth

如何避免仅在一次任务很长的情况下使用队列消息?

How can I avoid that consuming the queue message is done only once keeping in mind that the task is a long one?

推荐答案

首先:使用Azure存储队列，应该准备好所有操作成为幂等:如果您的队列项目被多次处理，每次都应发生相同的结果.我提出这个问题的原因:由于诸如角色实例崩溃/意外事件等意外事件，根本无法保证您一次处理消息(除非您检查消息的DequeueCount属性并相应地暂停处理).重新启动或您的队列项目处理代码做了意外的事情，例如引发异常.

First of all: Using Azure Storage queues, you should be prepared for all of your operations to be idempotent: In case your queue item is processed multiple times, the same result should happen each time. The reason I bring this up: There's simply no way to guarantee you'll process the message one time (unless you check the DequeueCount property of the message and halt processing accordingly), due to unexpected events such as your role instance crashing/rebooting or your queue item processing code doing something unexpected like throwing an exception.

下一步:可以以编程方式延长队列消息隐身超时时间.这可以通过队列api或通过sdk语言之一来完成.在C#中(类似这样-我没有测试过)，延长了另外一分钟:

Next: Queue message invisibility timeout can be programmatically extended. This can be done via the queue api or via one of the language sdk's. In c# (something like this - I didn't test this), extending an additional minute:

queueMessage.UpdateMessage(message, 
    TimeSpan.FromSeconds(60),
    MessageUpdateFields.Visibility);

您还可以在此过程中修改消息(也许作为代码提示，以使您知道5个报告中的哪一个已完成.这应该可以解决您的特定问题:如果重新处理了消息，则可以如果邮件已被修改为类似"process reports from 3-5"之类的内容，则不必处理所有五个报告.注意:您可以通过|组合MessageUpdateFields标志:

You can also modify the message along the way (maybe as a hint to your code, to let you know which of the 5 reports has been complete. This should help your specific issue: In the event the message gets reprocessed, you don't have to process all five reports if the message has been modified to say something like "process reports from 3-5"). Note: You can combine the MessageUpdateFields flags via |:

queueMessage.UpdateMessage(message, 
    TimeSpan.FromSeconds(0),
    MessageUpdateFields.Content);

最后:如果您担心要处理一批报告所花费的时间，也许要重新考虑一下为什么要在每封邮件中处理五个报告，而不是每封邮件处理一个报告.您始终可以批量读取队列消息. 这有点主观，因为实际上没有对与错的方法，但这只是您要考虑的事情.

Lastly: If you're concerned with the length of time taken to process a batch of reports, perhaps rethink why you're processing five reports in each message, vs. one report per message. You can always read queue messages in batches. This is getting a bit subjective, as there's really no right or wrong way to do it, but it's just something for you to think about.

这篇关于Azure存储队列-处理时间长的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！