删除嵌套的日期文件夹getdate()< 5日 [英] Delete nested date folder getdate() < 5 date

问题描述

DataLake中的文件夹具有嵌套日期文件夹结构

Folders in DataLake have nested date folder structure

2019 
    09
       29
       30
    10
       01
       02
       ..
       ..
       20

我已经使用活动对象for-each，GetMetaData，IfCondition和Delete编写了一个Datafactory

I have wrote an Datafactory using actives for-each , GetMetaData,IfCondition and Delete

{
    "name": "IterateEachADLSItem",
    "type": "ForEach",
    "dependsOn": [
        {
            "activity": "F_SAP",
            "dependencyConditions": [
                "Succeeded"
            ]
        }
    ],
    "userProperties": [],
    "typeProperties": {
        "items": {
            "value": "@activity('F_SAP').output.value",
            "type": "Expression"
        },
        "isSequential": false,
        "activities": [
            {
                "name": "GetPurgeFolderMetadata",
                "description": "GetPurgeFolderMetadata",
                "type": "GetMetadata",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "dataset": {
                        "referenceName": "DS_ADLS_FolderMetadata",
                        "type": "DatasetReference",
                        "parameters": {
                            "FolderPath": {
                                "value": "@concat(item().DestinationPath,item().SinkTableName,'/',item().LoadTypeName,'/',formatDateTime(adddays(utcnow(),-30),item().LoadIntervalFormat))",
                                "type": "Expression"
                            }
                        }
                    },
                    "fieldList": [
                        "exists"
                    ]
                }
            },
            {
                "name": "CheckPurgeFolderExists",
                "type": "IfCondition",
                "dependsOn": [
                    {
                        "activity": "GetPurgeFolderMetadata",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "expression": {
                        "value": "@activity('GetPurgeFolderMetadata').output.Exists",
                        "type": "Expression"
                    },
                    "ifTrueActivities": [
                        {
                            "name": "DeletePurgeFiles",
                            "type": "Delete",
                            "dependsOn": [],
                            "policy": {
                                "timeout": "7.00:00:00",
                                "retry": 0,
                                "retryIntervalInSeconds": 30,
                                "secureOutput": false,
                                "secureInput": false
                            },
                            "userProperties": [],
                            "typeProperties": {
                                "dataset": {
                                    "referenceName": "DS_ADLS_CopyDataSinkFile",
                                    "type": "DatasetReference",
                                    "parameters": {
                                        "SinkFilePath": {
                                            "value": "@concat(item().DestinationPath,item().SinkTableName,'/',item().LoadTypeName,'/',formatDateTime(adddays(utcnow(),-5),item().LoadIntervalFormat))",
                                            "type": "Expression"
                                        }
                                    }
                                },
                                "enableLogging": false,
                                "recursive": true
                            }
                        }
                    ]
                }
            }
        ]
    }
}

我可以在yyyy/mm/dd中删除确切日期之前的5天.如果我在2019/10/19执行，则可以删除2019/10/14

I am able to delete the exact 5 days before when it comes to date in the yyyy/mm/dd . if i execute on 2019/10/19, i am able to delete 2019/10/14

根据下面的语法，在确切的5个日期之前删除

Delete is fine based on below syntax for exact 5 dates before

''"value": "@concat(item().DestinationPath,item().SinkTableName,'/',item().LoadTypeName,'/',formatDateTime(adddays(utcnow(),-5),item().LoadIntervalFormat))",

如果存在现有文件夹，则无法在文件夹中查找每个月或更早的几天的早期日子.

I am not able to loop and find for earlier days in the folder for each month or earlier year provided there is an folder existing.

即: 例如在03/01/2020时如何删除上个月和前几年.

推荐答案

我认为您可以通过以下方法实现目标:

I think you could accomplish your goal by below approach:

1. 具有获取元数据"活动以列出根文件夹中的所有文件夹，并在GetMetadata中包含foreach活动.这应产生年份列表.
2. 包括条件活动以将当前年份与文件夹名称进行比较，如果该数目小于当前年份，则包括删除活动.
3. 如果以上条件不匹配，请包含另一个条件以检查其是否相等；如果是，请包含另一个Get Metadata活动
4. 以上获取元数据"活动将列出文件夹，该文件夹应产生月份列表并包括foreach活动.在月份中具有类似的逻辑，并包括另一个元数据活动以列出当月的日期.

1. Have Get Metadata activity to list all folders within root folder, and include foreach activity within GetMetadata.This should produce list of years.
2. Include Condition activity to compare the current year against folder's name and include delete activity,if that number is less than the current year.
3. If the above condition doesn't match, include another condition to check if it's equal and if yes, include another Get Metadata activity,
4. The above Get Metadata activity will list folders, which should produce months list and include foreach activity. Have similar logic for the months and include another metadata activity to list days for the current month.

1. 最后，最后一个获取元数据"活动将列出日期，并且当天数少于当前日期时，将包括删除"活动

希望有帮助， 菲克拉特

Hope that helps, Fikrat

这篇关于删除嵌套的日期文件夹getdate()< 5日的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！