获取元数据活动ADF V2 [英] Get Metadata Activity ADF V2

查看:148
本文介绍了获取元数据活动ADF V2的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

谁能解释我,ADF V2中新引入的获取元数据活动"有什么用途?

Can anyone explain me, what is the use of Get Metadata Activity that is newly introduced in ADF V2?

实际上,docs.microsoft.com中提供的信息不足以了解此活动的用途.

Actually, the information that is given in docs.microsoft.com isn't enough to understand the uses of this Activity.

推荐答案

获取元数据"活动的主要目的是:

Main purpose of the Get Metadata Activity is:

  • 验证任何数据的元数据信息
  • 在数据准备就绪/可用时触发管道

以下示例显示了如何使用获取元数据活动获取文件名和修改的时间戳从文件夹增量加载更改的文件:

The following example shows how to incrementally load changed files from a folder using the Get Metadata Activity getting filenames and modified Timestamp:

            {
                "name": "IncrementalloadfromSingleFolder",
                "properties": {
                    "activities": [
                        {
                            "name": "GetFileList",
                            "type": "GetMetadata",
                            "policy": {
                                "timeout": "7.00:00:00",
                                "retry": 0,
                                "retryIntervalInSeconds": 30,
                                "secureOutput": false
                            },
                            "typeProperties": {
                                "dataset": {
                                    "referenceName": "SrcLocalDir",
                                    "type": "DatasetReference"
                                },
                                "fieldList": [
                                    "childItems"
                                ]
                            }
                        },
                        {
                            "name": "ForEachFile",
                            "type": "ForEach",
                            "dependsOn": [
                                {
                                    "activity": "GetFileList",
                                    "dependencyConditions": [
                                        "Succeeded"
                                    ]
                                }
                            ],
                            "typeProperties": {
                                "items": {
                                    "value": "@activity('GetFileList').output.childItems",
                                    "type": "Expression"
                                },
                                "activities": [
                                    {
                                        "name": "GetLastModifyfromFile",
                                        "type": "GetMetadata",
                                        "policy": {
                                            "timeout": "7.00:00:00",
                                            "retry": 0,
                                            "retryIntervalInSeconds": 30,
                                            "secureOutput": false
                                        },
                                        "typeProperties": {
                                            "dataset": {
                                                "referenceName": "SrcLocalFile",
                                                "type": "DatasetReference"
                                            },
                                            "fieldList": [
                                                "lastModified"
                                            ]
                                        }
                                    },
                                    {
                                        "name": "IfNewFile",
                                        "type": "IfCondition",
                                        "dependsOn": [
                                            {
                                                "activity": "GetLastModifyfromFile",
                                                "dependencyConditions": [
                                                    "Succeeded"
                                                ]
                                            }
                                        ],
                                        "typeProperties": {
                                            "expression": {
                                                "value": "@and(less(activity('GetLastModifyfromFile').output.lastModified, pipeline().parameters.current_time), greaterOrEquals(activity('GetLastModifyfromFile').output.lastModified, pipeline().parameters.last_time))",
                                                "type": "Expression"
                                            },
                                            "ifTrueActivities": [
                                                {
                                                    "name": "CopyNewFiles",
                                                    "type": "Copy",
                                                    "policy": {
                                                        "timeout": "7.00:00:00",
                                                        "retry": 0,
                                                        "retryIntervalInSeconds": 30,
                                                        "secureOutput": false
                                                    },
                                                    "typeProperties": {
                                                        "source": {
                                                            "type": "FileSystemSource",
                                                            "recursive": false
                                                        },
                                                        "sink": {
                                                            "type": "BlobSink"
                                                        },
                                                        "enableStaging": false,
                                                        "dataIntegrationUnits": 0
                                                    },
                                                    "inputs": [
                                                        {
                                                            "referenceName": "SrcLocalFile",
                                                            "type": "DatasetReference"
                                                        }
                                                    ],
                                                    "outputs": [
                                                        {
                                                            "referenceName": "TgtBooksBlob",
                                                            "type": "DatasetReference"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    }
                                ]
                            }
                        }
                    ],
                    "parameters": {
                        "current_time": {
                            "type": "String",
                            "defaultValue": "2018-04-01T00:00:00Z"
                        },
                        "last_time": {
                            "type": "String",
                            "defaultValue": "2018-03-01T00:00:00Z"
                        }
                    },
                    "folder": {
                        "name": "IncrementalLoadSingleFolder"
                    }
                },
                "type": "Microsoft.DataFactory/factories/pipelines"
            }

另请参见最近更新的 查看全文

登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆