发布管道与集成运行时冲突 [英] Release pipeline conflict with integration runtime

问题描述

如果在数据工厂中定义了自托管的集成运行时，此问题与如何通过CI(在VSTS中)传播数据工厂有关.

This question relates to how to propagate a data factory through CI (in VSTS) if there is a self hosted Integration Runtime defined in the Data Factory.

我设置了3个环境-开发/开发/生产/生产每个都有自己的数据工厂.

I have a 3 environments set up - Dev / UAT / Prod each with their own data factory.

开发人员主持主协作分支.我正在使用VSTS从adf_publish分支中检索工件，并将模板部署到UAT(产品将在以后完成).我在此处 a>.

The Dev hosts the master collaboration branch. I am using VSTS to retrieve the artifacts from the adf_publish branch and deploying the template to UAT (prod will be done later). I followed much of what is in this guide here.

当使用自托管的集成运行时(IR)部署到空白UAT时，部署在UAT中的IR是来自dev(不是链接类型)的共享IR的副本，这会导致错误，因为使用了凭据IR所提供的信息将不正确.我希望如此，因为我们实际上只是在部署资源组模板的精确副本，只是覆盖了工厂名称，但是如果不使用自托管的IR虚拟机重新注册IR，则IR将无法工作.

When deploying to blank UAT with a self-hosted integration runtime (IR), the IR that is deployed in UAT is a copy of the shared IR from dev (not a linked type) and this causes an error since the credentials used by the IR will not be correct. I expect this since we are really just deploying an exact copy of the Resource Group template with just the factory name overridden however the IR will not work without it being re-credentialled with the self hosted IR VMs.

如果我在UAT环境中预先注册了一个链接的IR(已链接到dev IR)，则部署会因冲突而失败，因为资源组模板中的IR与我刚在UAT中创建的IR具有相同的名称. .如果名称不同-没有冲突，但是链接的服务将指向模板IR，而不是我为UAT创建的模板

If I pre-register a linked IR with the UAT environment (linked to the dev IRs), then the deployment fails with a conflict because an IR in the resource group template is the same name as the one I just created in UAT. If it is a different name - no conflict but the linked services will be pointing to the template IR and not the one I created for UAT

文档中有一条注释，指出所有平台上的IR运行时都应该相同，但我认为这不可能成立-其中一个(大概是源/开发)必须是共享类型，而其他链接和授权.

The docs have a note that says the IR runtime should be the same across all the platforms but I do not think this can be true - one of them (presumably the source/dev) must be a shared type and the others linked and authorized.

我可以看到的(未测试的)一个选项是让每个环境IR参考都是到实际IR的单独连接，但是然后需要某种方法来重写链接的服务以指向当前环境IR参考(通过模板参数覆盖?).在这种情况下，我们需要阻止模板IR的部署，因为它不会被使用，也将无法正常工作.

One option I could see (untested) is to have each environments IR reference be a separate connection to an actual IR but then there then needs to be some way of overriding the linked services to point to the current environments IR reference (by template parameter override?). In this scenario, we need to block the templates IR from being deployed as it won't be needed and won't work.

在这种情况下，有没有人成功地让CI工作?我的感觉是该文档是使用全球共享的IR编写的.或者，或者我需要更好地了解链接服务定义中自动集成"设置的目的.

Has anyone had success in getting CI working in this situation? My sense is the doc was written with the globally shared IR. Either that or I need to better understand the aim of Auto Integration setting in the linked services definition.

非常感谢. 马克.

推荐答案

更新 我认为该服务中存在一些错误，因此无法期待答案.如果我从发布的错误报告中看到解决方案，我将在此处发布更新此处.

Update I think there are a couple of bugs in the service so not expecting an answer. I'll post updates here if I see resolution from the bug report I have posted here for the dev group.

简而言之，这只会对您有影响

In a nutshell, this only affects you if

1. 您有一个自托管的集成运行时(IR)，并且
2. 您正尝试将模板从现有数据工厂部署到新的数据工厂(就像在Dev-> UAT-> Prod中一样)
3. 您已经定义了一个数据湖(ADL)链接服务，并使用了自托管的IR.

如果模板中有一个自托管的IR，则新部署的副本将不会在任何服务器上注册(新ADF链接或唯一)，因为该模板仅记录一个IR，而不会实例化一个IR.

If you have a self hosted IR in the template, the newly deployed copy will not be registered with any server (either linked or unique to the new ADF) as the template only records an IR, it does not instantiate one.

尽管可以在部署后的配置或脚本中解决此问题，但无法解决的是ADL中的依赖项.这是因为ADL链接服务希望使用IR加密服务主体，但是在模板部署时IR不存在(即未在服务器上配置且未处于活动状态).

While this can be fixed in post deployment config or scripting, what it can't fix is the dependency in ADL. This is because the ADL linked service wants to encrypt the service principal with the IR....but the IR does not exist at the time of template deployment (i.e. is not configured on a server and not active).

如果您选择托管服务身份作为ADL链接服务上的auth(而不是服务主体)作为身份验证，那就更好了，那么模板将无法部署，因为没有凭据可以加密，而且资源似乎希望对某些内容进行加密

It is no better if you select Managed service identity as the auth on the ADL linked service instead of service principal, then the template fails to deploy because there are no credentials to encrypt and it looks like the resource is expecting to encrypt something.

现在的解决方法是使用Azure托管的IR进行数据湖连接.对于我们来说不幸的是，这会导致安全问题，因为共享IR无法在我们的ADL Gen 1中列入白名单.

The work around right now is to use Azure hosted IR for datalake connections. Unfortunately for us this causes a security problem because shared IRs cannot be whitelisted in our ADL Gen 1.

我会及时通知你.

这篇关于发布管道与集成运行时冲突的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！