将数据从函数，事件中心，Azure数据资源管理器中提取? [英] Ingesting data from a function, to an event hub, to Azure Data Explorer?

问题描述

我有一些JSON数据进入IOT集线器，然后它触发了取消嵌套数据的功能.

I've got some JSON data coming into an IOT Hub, which then triggers a function to un-nest the data.

该函数将此数据发送到事件中心，然后应该由Azure Data Explorer根据我设置的映射来摄取数据.

The function sends this data to an Event Hub, and then the data is supposed to be ingested by Azure Data Explorer according to the mapping I've set up.

问题在于没有数据可以发送到数据浏览器；它将通过映射接收数据的唯一方法是将源设置为通过自定义路由接收信息的事件中心.

The problem is that no data makes it to the data explorer; the only way it will receive data with a mapping is by setting the origin as an event hub that is receiving information by custom routing.

是否可以通过IOT集线器->功能->事件集线器在数据资源管理器中提取数据?

Is it possible to ingest data in the data explorer by way of IOT hub -> function -> event hub?

用于取消嵌套并将数据转发到另一个事件中心的函数:

The function being used to un-nest and forward the data to another event hub:

module.exports = async function (context, eventHubMessages) {

    // receive message from IOT hub
    eventHubMessages.forEach((message, index) => {
        var devicename = message.deviceName;
        // timestamp comes in two different texts, find and store correct one
        var timestamp = (message.timestamp == null) ? message.timeStamp : message.timestamp;
        //context.log("Message: " + JSON.stringify(message));
        if (message.tags != null) {
            message.tags.forEach((tag, index) => {
                // for each tag, create new object
                var name = tag.Name;
                var value = tag.Value;
                var newObject = {
                                 "name":name,
                                 "value": value,
                                 "eventenqueuedutctime": timestamp,
                                 "devicename": devicename
                                }                
                // output message object to 'splitmessage-dev' event hub
                context.bindings.outputEventHubMessage = newObject
                context.log("Sent object: " + JSON.stringify(newObject));
            })
        }
    });

};

我可以确认其他事件中心正在接收此数据(已通过打印输出传入消息的其他功能进行了检查).

I can confirm the other event hub is receiving this data (checked with another function that prints the incoming messages).

映射如下:

'testTableMap' '[{"column":"name", "path":"$.name"}, 
{"column":"value", "path":"$.value"}, 
{"column":"eventenqueuedutctime", "path":"$.eventenqueuedutctime"},
{"column":"devicename", "path":"$.devicename"}]'

推荐答案

应该可以使用您提供的设置来摄取数据.

It should be possible to ingest data with the setup you provided.

由于未提取数据，因此您可能应该尝试诊断出数据被卡在哪里.

Since data isn't being ingested, you should probably try and diagnose where it gets stuck.

我将提供一些有助于调试的一般准则.

I'll provide some general guidelines that should help with debugging.

您可以检查EventHub监视并验证事件是否正在流动. 另外，我建议您从EventHub读取数据，以确保它看起来像您期望的那样. (似乎您已经这样做了)

You can check the EventHub monitoring and verify that events are flowing. Also, I would recommend reading the data from the EventHub just to make sure it looks like you would expect. (seems like you did this already)

这似乎不是这里的情况，因为您已经明确说明确实看到流入EventHub的事件，并且可以通过另一个函数成功读取它们.

This doesn't seem like the case here, as you have clarified that you do see events flowing into EventHub and you can read them successfully via another function.

尝试并从已配置的EventHub中获取数据

Try and ingest the data from the configured EventHub manually

// create table 
// I am assuming strings because other types can cause format errors, 
// so I would try strings before other specific types (dynamic, datetime)
.create table IngestionTest (name: string, value:string, eventenqueuedutctime:string, devicename:string)

// add mapping
.create table IngestionTest ingestion json mapping 'testTableMap' '[{"column":"name", "path":"$.name"}, {"column":"value", "path":"$.value"}, {"column":"eventenqueuedutctime", "path":"$.eventenqueuedutctime"},{"column":"devicename", "path":"$.devicename"}]'

// ingest data manually - replace with you actual data
.ingest inline into table IngestionTest with (jsonMappingReference='testTableMap') <| '{ "name" : "test", "value": { "some":"nested", "value": true}, "eventenqueuedutctime" : "2016-01-14", "devicename": "surface" }'

如果上述方法不起作用，则可以通过

If the above doesn't work, you can try and understand why by listing ingestion errors

.show ingestion failures

如果上述方法可行，则可以尝试使用其他数据类型.

If the above worked, you can try and play around with other data types.

您可以通过Azure门户检查的另一件事是ADX群集的指标.

One more thing you can check via Azure Portal, is metrics for the ADX cluster.

尝试进入Azure门户中的群集，然后在Metrics选项卡中可以选择两个指标来帮助您进行故障排除:

Try going to your cluster in Azure Portal, and in the Metrics tab you can pick two metrics that can help you troubleshoot:

Events Processed-使您了解ADX设法从EventHub读取了多少个事件.基本上，这使您知道数据源已正确配置.如果您没有看到任何事件，建议您建立一个新的来源.

Events Processed - Gives you a sense of how many events ADX managed to read from EventHub. This basicly lets you know that the Data Source is configured properly. If you see not events, I would suggest setting up a new source.

Ingestion Result-为您统计摄取状态(成功/失败)，这也可以帮助诊断失败.

Ingestion Result - Gives you a count on ingestion statuses (success / failure), which can also help diagnose failures.

这篇关于将数据从函数，事件中心，Azure数据资源管理器中提取?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！