Azure数据工厂-如何触发计划的/一次性管道? [英] Azure Data Factory - How can I trigger Scheduled/OneTime Pipelines?

问题描述

背景:我安排了一些运行中的管道，用于将数据从源复制到目标.它计划每天在特定时间运行.

Background : I have scheduled pipelines running for copying data from source to destination. This is scheduled to run daily at a specific time.

问题:管道的输入数据集是外部的，并且在特定的时间间隔不可用.这意味着复制活动将必须等到管道中提到的计划开始时间"才能开始.考虑到数据量，我不想在这里浪费时间.

Problem : The input dataset to the pipeline is external and not available at specific time intervals. This means the copy activity will have to wait until the Scheduled Start time mentioned in the Pipeline to kickoff. Considering the volume of data, I don't want to waste my time here.

要求::在任何给定时间，我都可以访问输入数据集可用的时间.有了这个，我想知道如何从C#触发ADF管道，尽管它计划仅在特定时间启动.

Requirement : At any given time I have access to the time when my input data set is available. With this in hand, I want to know how to trigger a ADF Pipeline from C# though its scheduled to start only at a specific time.

推荐答案

我遇到了同样的问题，我只需要在本地作业完成时才运行管道.为此，我修改了本地作业以启动管道的最后一步.我在这里写了一篇文章关于如何使用C#启动ADF管道. 此处是指向ADF开发人员参考的链接可能也有帮助.我也有一个示例此处(如果您有兴趣)如何从Azure Functions触发ADF管道.这使用的是与第一个示例相同的代码，但是我受益于在云中运行整个过程，并具有使用Azure函数调度程序的能力.

I ran into this same issue, I needed to run my pipeline only when a local job was completed. To do that I modified the local job to kick off the pipeline as its last step. I have a write up here on how to start an ADF pipeline with C#. Here is the link to the ADF developer reference which might also be helpful. I also have an example here on how to trigger ADF pipelines from Azure Functions, if you are interested. This is using the same code from the first example but I get the benefit of running the whole process in the cloud and the ability to use the azure function scheduler.

这是修改管道的相关方法.您需要根据希望切片运行的时间来更改开始日期和结束日期.

Here is the relevant method to modify the pipeline. You would need to change the start and end dates based on when you want the slice to run.

public void StartPipeline(string resourceGroup, string dataFactory, string pipelineName, DateTime slice)
    {
        var pipeline = inner_client.Pipelines.Get(resourceGroup, dataFactory, pipelineName);

        pipeline.Pipeline.Properties.Start = DateTime.Parse($"{slice.Date:yyyy-MM-dd}T00:00:00Z");
        pipeline.Pipeline.Properties.End = DateTime.Parse($"{slice.Date:yyyy-MM-dd}T23:59:59Z");
        pipeline.Pipeline.Properties.IsPaused = false;

        inner_client.Pipelines.CreateOrUpdate(resourceGroup, dataFactory, new PipelineCreateOrUpdateParameters()
        {
            Pipeline = pipeline.Pipeline
        });
    }

这篇关于Azure数据工厂-如何触发计划的/一次性管道?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！