在ADF V2中-如何为S3数据集动态地将日期("yyyyMMdd")附加到文件名中 [英] In ADF V2 - how to append date ("yyyyMMdd")) to filename dynamically for S3 dataset

问题描述

我目前正在自动执行ADFv2中的管道，其中源数据位于S3中.每天都会创建一个新文件，其结构为"data_20180829.csv"

I'm currently working to automate a pipeline in ADFv2 where the source data sits in S3. A new file is created daily and is structured "data_20180829.csv"

我尝试在复制数据活动的fileName字段中检测动态内容以实现此目的.但是，即使我尝试使用@ {concat('data _'，'20180829.csv')}之类的简单方法(应该解析为正确的值)，该源也会失败.

I have tried to instrument dynamic content to accomplish this in the fileName field of Copy Data Activity. However even when I try something as simple as @{concat('data_','20180829.csv')} (that should resolve to the correct value) the source fails.

是否可以查看动态内容将解决的内容?

Is there any way to see what the dynamic content will resolve to?

推荐答案

这应该只是在数据集中设置文件名表达式的问题，例如

This should just be a matter of setting the filename expression in the dataset, eg

请注意，设置是在数据集上完成的，而不是在复制"活动级别上进行的.还要注意，通过将utcnow函数与formatDateTime结合使用，可以使表达式更加动态，例如:

Note, the setting is done on the dataset not at the Copy activity level. Also note you can make your expression more dynamic by combining the utcnow function with formatDateTime, eg something like this:

@concat('data_',formatDateTime(utcnow(),'yyyMMdd_HHmmss'),'.csv')

请注意格式字符串的大小写.大写字母MM是月份的两位数格式. HH是24小时制的小时.格式字符串的完整列表为

Note carefully the case of the formatting strings. Capital MM is for month in a two-digit format. HH is for the hour in 24-hour format. The full list of formatting strings is here.

数据集的json看起来像这样:

The json for the dataset looks like this:

{
    "name": "DelimitedText3",
    "properties": {
        "linkedServiceName": {
            "referenceName": "linkedService2",
            "type": "LinkedServiceReference"
        },
        "annotations": [],
        "type": "DelimitedText",
        "typeProperties": {
            "location": {
                "type": "AzureBlobStorageLocation",
                "fileName": {
                    "value": "@concat('data_',formatDateTime(utcnow(),'yyyMMdd_HHmmss'),'.csv')",
                    "type": "Expression"
                },
                "container": "ang"
            },
            "columnDelimiter": ",",
            "escapeChar": "\\",
            "quoteChar": "\""
        },
        "schema": [
            {
                "type": "String"
            },
            {
                "type": "String"
            },
            {
                "type": "String"
            },
            {
                "type": "String"
            }
        ]
    }
}

这篇关于在ADF V2中-如何为S3数据集动态地将日期("yyyyMMdd")附加到文件名中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！