流分析:基于消息有效负载的动态输出路径 [英] Stream Analytics: Dynamic output path based on message payload

问题描述

我正在研究一个IoT分析解决方案，该解决方案使用在Azure IoT中心发射的Avro格式的消息，并且(希望)使用Stream Analytics将消息存储在Data Lake和Blob存储中.一个关键要求是，Avro容器在存储中的外观必须与呈现给IoT Hub时的外观完全相同，以使下游消费者受益.

I am working on an IoT analytics solution which consumes Avro formatted messages fired at an Azure IoT Hub and (hopefully) uses Stream Analytics to store messages in Data Lake and blob storage. A key requirement is the Avro containers must appear exactly the same in storage as they did when presented to the IoT Hub, for the benefit of downstream consumers.

我在Stream Analytics中遇到了限制，无法对单个文件的创建进行精细控制.设置新的输出流路径时，我只能在路径前缀中提供日期/日期和小时，导致每小时生成一个文件，而不是每个收到的消息生成一个文件.客户需要为每个设备使用单独的Blob容器，并为每个事件需要单独的Blob.同样，Data Lake要求至少规定了一个合理的命名约定，该约定由设备描述，并为每个事件提取了单独的文件.

I am running into a limitation in Stream Analytics with granular control over individual file creation. When setting up a new output stream path, I can only provide date/day and hour in the path prefix, resulting in one file for every hour instead of one file for every message received. The customer requires separate blob containers for each device and separate blobs for each event. Similarly, the Data Lake requirement dictates at least a sane naming convention that is delineated by device, with separate files for each event ingested.

是否有人成功配置Stream Analytics在每次从输入中弹出消息时创建一个新文件?这是硬性产品限制吗?

Has anyone successfully configured Stream Analytics to create a new file every time it pops a message off of the input? Is this a hard product limitation?

推荐答案

Stream Analytics确实面向有效处理大型流. 对于您的用例，您需要一个附加组件来实现您的自定义逻辑.

Stream Analytics is indeed oriented for efficient processing of large streams. For your use case, you need an additional component to implement your custom logic.

Stream Analytics可以输出到Blob，事件中心，表存储或服务总线.另一个选择是使用新的

Stream Analytics can output to Blob, Event Hub, Table Store or Service Bus. Another option is to use the new Iot Hub Routes to route directly to an Event Hub or a Service Bus Queue or Topic.

您可以在此处编写 Azure函数(或从Blob或表存储，自定义数据工厂活动)，并使用

From there you can write an Azure Function (or, from Blob or Table Storage, a custom Data Factory activity) and use the Data Lake Store SDK to write files with the logic that you need.

这篇关于流分析:基于消息有效负载的动态输出路径的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！