ADF获取数据和比较 [英] ADF get data and comparison

问题描述

Hello Experts，

Hello Experts,

我们有一个场景，我们需要从azure中的两个现有数据源获取一些数据，然后在这些数据之间进行比较并最终复制他们到我们的数据湖。我认为这可以通过ADF和Pipeline达成。但是，如果有可能并且可以使用哪些活动，我没有任何
的想法。 ADF有没有办法做到这一点？ 

We have a scenario in which we need to get some data from two existing data source in azure and then do a comparison between those data and finally copy them to our data lake. I thought this can be reach by ADF and Pipeline. However, I don't have any idea if it is possible and which activities can be used. Is there any way to do that by ADF? 

谢谢

Shayan

推荐答案

Hi Shayan，

Hi Shayan,

复制活动不适用于多个输入或输出。它只能执行1到1的复制。



我建议使用单独的复制活动（每个文件1个）将1个管道复制到某种Azure存储中。然后具有第二下游流管道，其具有"变换"管道。读取和合并/连接文件的活动，以产生单个输出
。这里提供的不同转换活动包括：

The copy activity doesn't work with multiple inputs or outputs. It can only perform a 1 to 1 copy.

I suggest having 1 pipeline copying both files into some sort of Azure storage using separate copy activities (1 per file). Then have a second down stream pipeline that has a "transform" activity to read and merge/concatenate the files to produce a single output. The different transform activities available here are :

• 自定义活动，要了解更多信息，请参阅
  此文档。 
• 如果Sink是SQL DB或SQL数据仓库，也可以使用存储过程。要阅读有关在ADF中运行存储过程的更多信息，请参阅
  
  此文档。
• 如果您的接收器是Azure Data Lake Store，则还可以在运行后运行U-SQL活动。

• Custom activities, To read more about it please refer this doc. 
• You can also use a Stored Procedure if your Sink is a SQL DB or a SQL Data Warehouse. To read more about running a Stored Procedure in ADF, please refer this doc.
• If your sink is Azure Data Lake Store, you can also run a U-SQL activity post run.

希望这会有所帮助。

这篇关于ADF获取数据和比较的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！