发送到Microsoft Azure EventHubs时如何使用客户端事件批处理功能 [英] How to use client-side event batching functionality while Sending to Microsoft Azure EventHubs
问题描述
我正在处理EventHub的高吞吐量应用程序.根据文档，要从单个发件人获得非常高的吞吐量，则需要客户端批处理(每个事件不得超过256 KB限制).
I'm dealing with a high throughput application of EventHub. According to the documentation, in order to achieve very high throughput from a single sender, then client-side batching is required (without exceeding the 256 KB limit per event).
使用Service Bus代理消息传递提高性能的最佳做法建议通过客户端批处理来提高性能.它描述了可用于队列或主题客户端的客户端批处理，这可以将消息的发送延迟一定的时间，然后以单个批处理的方式传输消息.
Best Practices for performance improvements using Service Bus brokered messaging suggests Client-side batching for achieving performance improvements. It describes client-side batching is available for queue or topic clients, which enables delaying the sending of messages for a certain period of time, then it transmits the messages in a single batch.
EventHub客户端中可以进行客户端批处理吗?
Is client-side batching available in the EventHub client?
推荐答案
ShortAns :EventHubs旨在支持非常高的吞吐量场景-客户端批处理是实现此目的的关键功能之一. API是`EventHubClient.SendBatch(IEnumerable ).
ShortAns: EventHubs is designed to support very-high thruput scenarios - Client-side batching is one of the Key features to enable this. API is `EventHubClient.SendBatch(IEnumerable).
长篇故事:
您找到的链接:改善性能的最佳做法使用Service Bus中介消息传递适用于ServiceBus队列&主题-使用称为SBMP的Microsoft专有协议，并且不是开放标准.我们在该协议中实现了BatchFlushInterval.这是前一段时间(2010年左右的猜测)-其中 Amqp 协议尚未标准化.当我们开始构建Azure EventHubs服务时- Amqp 作为我们的事件中心一流协议. BatchFlushInterval在EventHubs中不起作用(
Long Story:
The link that you found: Best Practices for performance improvements using Service Bus brokered messaging applies to ServiceBus Queues & Topics - which uses a Microsoft Proprietary protocol called - SBMP - and is not an Open Standard. We implemented BatchFlushInterval in that Protocol. This was a while back (guess around 2010) - where Amqp protocol wasn't standardized yet. When we started building Azure EventHubs service - Amqp is the new Standard protocol for implementing performant messaging solutions and hence, we used Amqp as our first-class protocol for Event Hubs. BatchFlushInterval doesn't have any effect in EventHubs (Amqp).
EventHubClient将您需要发送到EventHub的每个原始事件转换为AmqpMessage(请参阅( Amqp协议规范).
为了做到这一点，按照协议，它向每个消息添加了几个额外的字节.可以使用属性- SerializedSizeInBytes .
EventHubClient translates every raw event that you need to send to EventHub into AmqpMessage (refer to Messaging section in the (Amqp Protocol Specification).
In order to do that, as per the protocol, it adds few extra bytes to each Message. The estimated Size of each Serialized EventData (to AmqpMessage) can be found using the property - EventData SerializedSizeInBytes.
在这种背景下，您将遇到以下情况:实现高吞吐量的最佳方法-使用 EventHubClient.SendBatch(IEnumerable<EventData>) api.此Api的合同是-调用SendBatch之前-调用方需要确保此批消息的序列化大小不超过256k.在内部，此API将IEnumerable<EventData>转换为1个单个AmqpMessage并发送到EventHub服务.截至2016年4月25日，EventHubs服务对1个单一AmqpMessage施加的限制为256k.另外，还有一个详细信息-将EventData的列表转换为单个AmqpMessage时-EventHubClient需要将一些信息升级到BatchMessage标头中-这对于批处理中的所有这些消息都是通用的(例如partitionKey之类的信息) ).此信息.保证最大为6k.
因此，总的来说，呼叫者需要跟踪IEnumerable<EventData>中所有EventData的总大小，并确保其小于250k .
With that background, coming to your scenario: Best way, to achieve very high-thruputs - is to use EventHubClient.SendBatch(IEnumerable<EventData>) api. The contract of this Api is - before invoking SendBatch - the caller need to make sure the Serialized Size of this Batch of messages doesn't exceed 256k. Internally, this API converts the IEnumerable<EventData> into 1 Single AmqpMessage and sends to EventHub Service. The limit on 1 single AmqpMessage imposed by EventHubs service as-of 4-25-2016 is 256k. Plus, one more detail - when the list of EventData are translated to a Single AmqpMessage - EventHubClient needs to promote some information into the BatchMessage header - which is common for all of those messages in the batch(info like partitionKey). This info. is guaranteed to be a max of 6k.
So, all-in-all, the caller need to keep track of the aggregate size of all EventData in the IEnumerable<EventData> and make sure that this falls below 250k.
编辑日期:2017年9月14日
构造EventData批处理不再涉及任何猜测工作.从EventHubClient.CreateBatch API获取Empty EventDataBatch，然后使用TryAdd(EventData) api添加事件以构造批处理.
并且，最后使用EventDataBatch.ToEnumerable()获取基础事件以传递给EventHubClient.Send() API.
EDIT ON 09/14/2017
WE added EventHubClient.CreateBatch API to support this scenario.
There is no more guess work involved in constructing a Batch of EventDatas. Get an Empty EventDataBatch from EventHubClient.CreateBatch API and then use TryAdd(EventData) api to add events to construct the Batch.
And, finally use EventDataBatch.ToEnumerable() to get the underlying events to pass to the EventHubClient.Send() API.
这篇关于发送到Microsoft Azure EventHubs时如何使用客户端事件批处理功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
