Azure数据工厂V2复制活动-保存所有已复制文件的列表 [英] Azure Data Factory V2 Copy Activity - Save List of All Copied Files

问题描述

我有一些管道，可以将文件从本地复制到不同的接收器，例如本地和SFTP.我想保存每次运行中复制的所有文件的列表以进行报告.

I have pipelines that copy files from on-premises to different sinks, such as on-premises and SFTP. I would like to save a list of all files that were copied in each run for reporting.

我尝试使用Get Metadata和For Each，但不确定如何将输出保存到平面文件甚至数据库表中.

I tried using Get Metadata and For Each, but not sure how to save the output to a flat file or even a database table.

或者，可以对复制到Data Factory日志中某处的对象列表进行优化吗?

Alternatively, is it possible to fine the list of object that are copied somewhere in the Data Factory logs?

谢谢

推荐答案

更新:

项目: @activity('Get Metadata1').output.childItems

如果要记录源文件名，可以.如您所说，我们需要使用Get Metadata和For Each活动.
我创建了一个测试，以将复制"活动的源文件名保存到SQL表中.

If you want record the source file names, yes we can. As you said we need to use Get Metadata and For Each activity.
I've created a test to save the source file names of the Copy activity into a SQL table.

1. 众所周知，我们可以通过获取元数据"活动中的子项获取文件列表.Get Metadata1活动的数据集指定包含多个文件的容器. test 容器中的文件列表如下:
   

1. As we all know, we can get the file list via Child items in Get metadata activity. The dataset of Get Metadata1 activity specify the container which contains several files. The list of file in test container is as follows:
   

在ForEach活动的内部中，我们可以遍历此数组.我设置了一个名为 Copy-Files 的Copy活动，以将文件从源复制到目标.

At inside of the ForEach activity, we can traverse this array. I set a Copy activity named Copy-Files to copy files from source to destnation.

@item().name 表示 test 容器中的每个文件.我键入动态内容 @item().name 以指定文件名.然后它将依次在测试容器中传递文件名.这是要分批执行复制任务，每批将传递一个要复制的文件名.这样我们以后可以将每个文件名记录到数据库表中.

@item().name represents every file in the test container. I key in the dynamic content @item().name to specify the file name. Then it will sequentially pass the file names in the test container. This is to execute the copy task in batches, each batch will pass in a file name to be copied. So that we can record each file name into the database table later.

然后，我设置另一个复制"活动，以将文件名保存到SQL表中.在这里，我使用的是Azure SQL，并且创建了一个简单的表.

Then I set another Copy activity to save the file names into a SQL table. Here I'm using Azure SQL and I've created a simple table.

create table dbo.File_Names(
    Copy_File_Name varchar(max)
);

1. 此

然后我们可以将文件名存储到SQL表中.
选择先前创建的表.

Then we can sink the file names into the SQL table.
Select the table which created previously.

运行调试后，可以看到所有文件名都保存到表中.

After I run debug, I can see all the file names are saved into the table.

如果要添加更多信息，可以引用

If you want add more infomation, you can reference the post I maintioned previously.

这篇关于Azure数据工厂V2复制活动-保存所有已复制文件的列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！