在复制数据中从源到接收器丢失数据 [英] Loosing data from Source to Sink in Copy Data

问题描述

在我的MS Azure数据工厂中，我有一个与嵌套JSON数据集的rest API连接.源预览数据"显示所有数据.(来自在线商店的7个订单)

在活动复制数据"中，是菜单选项卡映射"，在其中我将JSON字段与接收器SQL表列进行映射.如果我在集合引用"下选择了无"，则将复制所有7个订单.但是，如果我需要嵌套的元数据，则可以在集合引用"中选择meta字段，然后以多个顺序行获取嵌套数据，每个顺序行都有一个元数据点，但是我只能从1个顺序获取数据，而不是7

我认为我有问题的原因.嵌套元数据中的字段之一是字符串和数组.但我仍然没有解决办法

我的解决方法是使用Azure Blob存储进行过渡， REST API --->Azure Blob存储--->您的接收器数据集.在Blob存储数据集内部，您可以使用交叉应用嵌套的JSON数组设置来展平传入的JSON数据:

您可以参考此博客了解此功能.然后，您可以将展平的数据复制到目标位置.

In my MS Azure datafactory, I have a rest API connection to a nested JSON dataset. The Source "Preview data" shows all data. (7 orders from the online store)

In the "Activity Copy Data", is the menu tab "Mapping" where I map JSON fields with the sink SQL table columns. If I under "Collection Reference" I select None, all 7 orders are copied over. But if I want the nested metadata, I select the meta field in "Collection Reference", then I get my nested data, in multiple order lines, each with a one metadata point, but I only get data from 1 order, not 7

I think I have a reason for my problem. One of the fields in the nested meta data, is both a string and array. But I still don't have a solution

sceen shot of meta data

解决方案

Your sense is right,it caused by your nested structure meta data. Based on the statements of Collection Reference property:

If you want to iterate and extract data from the objects inside an array field with the same pattern and convert to per row per object, specify the JSON path of that array to do cross-apply. This property is supported only when hierarchical data is source.

same pattern is key point here, I think. However, your data inside metadata array are not same as your screenshot.

My workaround is using Azure Blob Storage to make a transition, REST API ---> Azure Blob Storage--->Your sink dataset. Inside Blob Storage Dataset, you could flatten the incoming JSON data with Cross-apply nested JSON array setting:

You could refer to this blog to learn about this feature. Then you could copy the flatten data into your destination.

这篇关于在复制数据中从源到接收器丢失数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！