仍在运行时停止运行Azure Data Factory Pipeline [英] Stop running Azure Data Factory Pipeline when it is still running
问题描述
我有一个Azure Data Factory Pipeline.每隔5分钟设置一次我的触发器.
有时我的管道需要5分钟以上才能完成工作.在这种情况下，触发器再次运行并创建我的管道的另一个实例，并且同一管道的两个实例使我的ETL出现问题.
我怎么能确定管道中只有一个实例同时运行?
I have a Azure Data Factory Pipeline. My trigger has been set for every each 5 minutes.
Sometimes my Pipeline takes more than 5 mins to finished its jobs. In this case, Trigger runs again and creates another instance of my Pipeline and two instances of the same pipeline make problem in my ETL.
How can I be sure than just one instance of my pipeline runs at time?
如您所见，我的管道有多个实例在运行
As you can see there are several instances running of my pipelines
推荐答案
我想到的几个选项:
OPT 1
在管道活动中指定5分钟超时:
Specify 5 min timeout on your pipeline activities:
https://docs.microsoft.com/zh-CN/azure/data-factory/concepts-pipelines-activities https://docs.microsoft. com/en-us/azure/data-factory/concepts-pipelines-activities#activity-policy
OPT 2
1)创建1行1列的sql RunStatus表:1将是我们的已完成"，0-是正在运行"状态
1) Create a 1 row 1 column sql RunStatus table: 1 will be our "completed", 0 - "running" status
2)在管道的末尾添加一个存储过程活动，该活动会将该位设置为1.
2) At the end of your pipeline add a stored procedure activity that would set the bit to 1.
3)在管道的开头，添加一个查找活动以读取该位.
3) At the start of your pipeline add a lookup activity to read that bit.
4)然后在条件活动中使用此查询的输出:
4) The output of this lookup will then be used in if condition activity:
- 如果1-启动管道的工作，但在此之前添加另一个存储过程活动以将我们的状态位设置为0.
- 如果为0，则取决于项目的详细信息:不执行任何操作，添加等待活动，发送电子邮件等.
要充分利用此选项，您可以将表转换为日志，每次成功运行后都会在其中添加带有开始和结束时间的新行(在开始新运行之前，您可以检查是否先前运行已结束时间).拥有此日志可能会帮助您收集有关运行管道需要花费多少的数据，也许会添加更多资源或增加运行之间的间隔.
To make a full use of this option, you can turn the table into a log, where the new line with start and end time will be added after each successful run (before initiating a new run, you can check if the previous run had the end time). Having this log might help you gather data on how much does it take to run your pipeline and perhaps either add more resources or increase the interval between the runs.
OPT 3
监视使用SDK运行的管道(尚未尝试过，因此仅是为了指导您): https://docs.microsoft.com/zh-cn/天蓝色/数据工厂/监控程序
Monitor the pipeline run with SDKs (have not tried that, so this is just to possibly direct you): https://docs.microsoft.com/en-us/azure/data-factory/monitor-programmatically
希望您至少可以使用其中之一
Hopefully you can use at least one of them
这篇关于仍在运行时停止运行Azure Data Factory Pipeline的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
