在数据湖事件触发器上执行ADFV2管道 [英] ADFV2 pipeline execution on a data lake event trigger

问题描述

我们有大约190个小时使用情况文件，它们需要在24小时内到达数据湖，然后才能启动我们的管道，该管道从分析活动开始.我们已经在预期的预计时间上在调度程序上运行了此管道 所有文件都已到达，但并非总是会发生，因此我们需要为丢失的文件重新运行分片.

是否有一种更有效的方法来处理此问题，而不是按计划进行管道传输，而是在所有文件都已到达数据湖的事件中触发它.

输入的TIA！

解决方案

你好

我知道您的文件位于Data Lake上，但是您是否考虑过基于Blob存储的基于事件的触发器:

https://docs.microsoft.com/zh-cn/azure/data-factory/how-to-create-event-trigger

We have about 190 hourly usage files that need to arrive in the data lake in a 24 hour period before we can kick off our pipeline which starts off with an analytics activity. We have had this pipeline run on a scheduler on an estimated time of when we expect all files to have arrived but doesn't always happen so we would need to re-run the slices for the missing files. 

Is there a more efficient way to handle this and not have the pipeline on a schedule and have it triggered by the event that all files have arrived in the datalake.

TIA for input!

解决方案

Hello,

I know your files are on Data Lake, but have you considered event-based triggers based on blob storage:

https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger

这篇关于在数据湖事件触发器上执行ADFV2管道的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！