Azure数据工厂|从SFTP到Blob的增量数据加载 [英] Azure data factory | incremental data load from SFTP to Blob

问题描述

我创建了一个(一次性运行)DF(V2)管道，以将文件(.lta.gz)从SFTP服务器加载到一个天蓝色的blob中以获取历史数据. 做工精美. 每天，SFTP服务器上都会有几个新文件(无法操纵或删除).因此，我想创建一个增量加载管道，该管道每天检查新文件-如果是--->复制新文件.

I created a (once run) DF (V2) pipeline to load files (.lta.gz) from a SFTP server into an azure blob to get historical data. Worked beautifully. Every day there will be several new files on the SFTP server (which cannot be manipulated or deleted). So I want to create an incremental load pipeline which checks daily for new files - if so ---> copy new files.

有人对我有什么秘诀吗?

Does anyone have any tips for me how to achieve this?

推荐答案

感谢使用Data Factory！

Thanks for using Data Factory!

要在SFTP服务器上增量加载新生成的文件，可以利用GetMetadata活动来检索LastModifiedDate属性: https://docs.microsoft. com/en-us/azure/data-factory/control-flow-get-metadata-activity

To incrementally load newly generated files on SFTP server, you can leverage the GetMetadata activity to retrieve the LastModifiedDate property: https://docs.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity

基本上，您编写的管道包含以下活动:

Essentially you author a pipeline containing the following activities:

• getMetadata(返回给定文件夹下的文件列表)
• ForEach(遍历每个文件)
• getMetadata(为给定文件返回lastModifiedTime)
• IfCondition(将lastModifiedTime与触发器WindowStartTime进行比较)
• 复制(将文件从源复制到目标)

使用Data Factory构建数据集成流程很有趣！

Have fun building data integration flows using Data Factory!

这篇关于Azure数据工厂|从SFTP到Blob的增量数据加载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！