VSTS 构建管道:测试连接到 Azure Key Vault 失败 [英] VSTS Build Pipeline: Test fails connecting to Azure Key Vault

问题描述

我正在尝试使用 VSTS(现在是 Azure DevOps)来执行 CI/CD 管道.对于我的构建管道，我有一个非常基本的设置，包括执行还原、构建、测试和发布步骤.

对于我的测试步骤，我将它设置为运行两个测试项目 - 一个单元测试项目和一个集成测试项目.我设置了 Key Vault 访问策略以提供对我自己和 Azure Devops 的访问.当我使用 Visual Studio 在本地运行我的测试时，因为我登录到可以访问 Azure Key Vault 的同一个帐户，我可以运行测试而不会出现任何错误.

我的应用程序配置为使用以下设置访问密钥保管库:

 public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>WebHost.CreateDefaultBuilder(args).ConfigureAppConfiguration((ctx, builder) =>{var keyVaultEndpoint = GetKeyVaultEndpoint();if (!string.IsNullOrEmpty(keyVaultEndpoint)){var azureServiceTokenProvider = new AzureServiceTokenProvider();var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));builder.AddAzureKeyVault(keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());}}).UseStartup();

当我运行构建管道时，我使用托管 VS2017 实例来构建我的项目.除了尝试访问密钥保管库的集成测试失败之外，一切都在运行.我正在使用以下软件包:

• Microsoft.Azure.Services.AppAuthentication - 易于获取服务到 Azure 服务身份验证方案的访问令牌.
• Microsoft.Azure.KeyVault - 包含与 Key Vault 交互的方法.
• Microsoft.Extensions.Configuration.AzureKeyVault - 包含
  Azure Key Vault 的 IConfiguration 扩展

我遵循了本教程

以下是错误的堆栈跟踪:

 2018-10-16T00:37:04.6202055Z 测试运行 D:a1sSGIntegrationTestsinRelease
etcoreapp2.1SGIntegrationTests.dll(.NETCoreApp,Version=v2.1)2018-10-16T00:37:05.3640674Z Microsoft (R) 测试执行命令行工具版本 15.8.02018-10-16T00:37:05.3641588Z 版权所有 (c) Microsoft Corporation.版权所有.2018-10-16T00:37:05.3641723Z2018-10-16T00:37:06.8873531Z 开始测试执行，请稍候...2018-10-16T00:37:51.9955035Z [xUnit.net 00:00:40.80] SGIntegrationTests.HomeControllerShould.IndexContentTypeIsTextHtml [失败]2018-10-16T00:37:52.0883568Z SGIntegrationTests.HomeControllerShould.IndexContentTypeIsTextHtml 失败2018-10-16T00:37:52.0884088Z 错误消息:2018-10-16T00:37:52.0884378Z Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException:参数:连接字符串:[未指定连接字符串]，资源:https://vault.azure.net，权限:https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937.异常消息:尝试了以下 3 种方法来获取访问令牌，但都没有奏效.2018-10-16T00:37:52.0884737Z 参数:连接字符串:[未指定连接字符串]，资源:https://vault.azure.net，权限:https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937.异常消息:尝试使用托管服务标识获取令牌.无法获取访问令牌.MSI ResponseCode: BadRequest, Response: {"error":"invalid_request","error_description":"Identity not found"}2018-10-16T00:37:52.0884899Z 参数:连接字符串:[未指定连接字符串]，资源:https://vault.azure.net，权限:https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937.异常消息:尝试使用 Visual Studio 获取令牌.无法获取访问令牌.在C:UsersVssAdministratorAppDataLocal.IdentityServiceAzureServiceAuth	okenprovider.json"中找不到 Visual Studio 令牌提供程序文件2018-10-16T00:37:52.0885142Z 参数:连接字符串:[未指定连接字符串]，资源:https://vault.azure.net，权限:https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937.异常消息:尝试使用 Azure CLI 获取令牌.无法获取访问令牌.返回令牌的过程花费了太长时间.2018-10-16T00:37:52.0885221Z2018-10-16T00:37:52.0885284Z 堆栈跟踪:2018-10-16T00:37:52.0885349Z 在 Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.GetAccessTokenAsyncImpl(字符串权限、字符串资源、字符串范围)2018-10-16T00:37:52.0885428Z 在 Microsoft.Azure.KeyVault.KeyVaultCredential.PostAuthenticate(HttpResponseMessage 响应)2018-10-16T00:37:52.0885502Z 在 Microsoft.Azure.KeyVault.KeyVaultCredential.ProcessHttpRequestAsync(HttpRequestMessage 请求，CancellationToken 取消令牌)2018-10-16T00:37:52.0886831Z 在 Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancelationToken)2018-10-16T00:37:52.0886887Z 在 Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretsAsync(IKeyVaultClient 操作，字符串 vaultBaseUrl，Nullable`1 maxresults，CancellationToken 取消令牌)2018-10-16T00:37:52.0886935Z 在 Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()2018-10-16T00:37:52.0887000Z 在 Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()2018-10-16T00:37:52.0887045Z 在 Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 提供商)2018-10-16T00:37:52.0887090Z 在 Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()2018-10-16T00:37:52.0887269Z 在 Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException&hostingStartupErrors)2018-10-16T00:37:52.0887324Z 在 Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()2018-10-16T00:37:52.0887371Z 在 Microsoft.AspNetCore.TestHost.TestServer..ctor(IWebHostBuilder builder, IFeatureCollection featureCollection)2018-10-16T00:37:52.0887433Z 在 Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateServer(IWebHostBuilder builder)2018-10-16T00:37:52.0887477Z 在 Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()2018-10-16T00:37:52.0887525Z 在 Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)

<小时>

更新

我只找到了 1 个与此问题相关的帖子:

为您的新 AD 应用提供对密钥保管库的访问权限.进入您的 Key Vault 访问策略并添加您在 AD 中创建的具有对您的机密的读取访问权限的应用程序.

在我的 Program.cs 文件中修改了对 AzureServiceTokenProvier() 的调用，如下所示:

 var azureServiceTokenProvider = new AzureServiceTokenProvider("connectionString={您的密钥保管库端点};RunAs=App;AppId={您在 Azure AD 中设置的应用程序 ID};TenantId={您的 azure 订阅};AppKey={您的客户密钥}")

请注意，您的客户端密钥的格式必须正确.应用程序注册(预览)会生成一个随机密钥.有时此键在连接字符串中不起作用(由于格式不正确而引发错误).尝试在非预览版应用注册中生成您自己的密钥，或者生成一个新密钥并重试.

之后，我能够在我的构建管道中成功运行我的集成测试，并在 Azure 中为我的 Web 应用程序创建一个版本.我对这种方法并不满意，因为虽然它有效，但它在代码本身中暴露了一个秘密值.由于上述方法，不需要开启管理服务标识.我觉得这在这方面非常糟糕.

必须有比这更好的方法.一种选择是不在构建管道中运行集成测试.不确定这是否是正确的方法.我仍然希望有人能够为此提供更好的方法或解释我的方法是否可以使用.

解决方案

您不应在 Azure DevOps Pipelines 构建中对 Azure KeyVault 进行身份验证的集成测试，因为您使用的是 Azure DevOps 默认托管代理.

默认情况下，Azure DevOps Pipelines 使用基本的默认托管代理，并且您的 Azure 订阅无法访问这些托管代理.这些并不奇怪，因为这些托管代理是所有常见构建需求的通用代理，包括构建/编译、运行单元测试、获取测试覆盖率，并且所有这些任务都没有其他附加功能，例如拥有 ActiveDirectory、数据库和其他对其他方的实际身份验证/请求，例如对任何 Azure Keyvault 的身份验证.因此，默认情况下这些代理未在您的 Azure 订阅中注册.

如果您希望针对这些特殊需求进行成功的集成测试，您必须为 Azure DevOps Pipelines 构建和发布创建您自己的代理.因此，除了创建您自己的代理并配置您的 Azure DevOps 以使用您自己的代理之外，没有其他方法可以强制 Azure DevOps 默认代理运行您的 KeyVault 身份验证测试.

要创建您自己的代理，请参阅 Microsoft 的此文档:

https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=vsts#install

2018 年 10 月 29 日更新:

为了更清楚，我还回复了您的更新 3"解决方法.当 Microsoft 更新 Azure DevOps 的默认托管代理时，无法保证您的解决方法会很好地工作.因此，我还需要补充一点:集成测试依赖于 Azure DevOps Pipelines 构建领域之外的其他方，例如连接到数据库服务器或使用外部身份验证(即使在 Azure KeyVault 上)，这不是一个好的做法.您的 CI，尤其是当您使用 Microsoft 的默认托管代理时.

不仅由于身份验证配置无效而容易出错，而且不能保证对默认托管代理的进一步更新将保证您的第三方逻辑测试能够正常工作.

I am trying to use VSTS (now Azure DevOps) to do a CI/CD pipeline. For my build pipeline, I have a very basic setup involving doing a restore, build, test, and publish steps.

For my test step, I have it setup to run two test projects - one unit test project and one integration test project. I have my Key Vault access policy setup to provide access to both myself and Azure Devops. When I run my tests locally using visual studio, as I am logged into the same account which has access to azure key vault, I can run the tests without any errors.

My application is configured to access key vault using below setup:

 public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((ctx, builder) =>
            {
                var keyVaultEndpoint = GetKeyVaultEndpoint();

                if (!string.IsNullOrEmpty(keyVaultEndpoint))
                {
                    var azureServiceTokenProvider = new AzureServiceTokenProvider();
                    var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
                    builder.AddAzureKeyVault(keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());
                }
            }
        )
            .UseStartup<Startup>();

When I run the build pipeline, I am using a Hosted VS2017 instance to build my project. Everything is running except the integration tests which try and access the key vault fail. I am using the following packages:

• Microsoft.Azure.Services.AppAuthentication - makes it easy to fetch access tokens for Service-to-Azure-Service authentication scenarios.
• Microsoft.Azure.KeyVault - contains methods for interacting with Key Vault.
• Microsoft.Extensions.Configuration.AzureKeyVault - contains
  IConfiguration extensions for Azure Key Vault

I followed this tutorial https://docs.microsoft.com/en-us/azure/key-vault/tutorial-web-application-keyvault to setup the key vault and integrate it into my app.

I am merely trying to get my build to work by making sure both the unit and integration tests pass. I am not deploying it to an app service yet. The unit tests run without any issues as I am mocking the various services. My integration test is failing with below error messages. How do I get my test access to the key vault? Do I need to add any special access policies to my key vault for the hosted VS2017 build? Not sure what to do as I don't see anything that stands out.

Below is the stack trace for the error:

    2018-10-16T00:37:04.6202055Z Test run for D:a1sSGIntegrationTestsinRelease
etcoreapp2.1SGIntegrationTests.dll(.NETCoreApp,Version=v2.1)
    2018-10-16T00:37:05.3640674Z Microsoft (R) Test Execution Command Line Tool Version 15.8.0
    2018-10-16T00:37:05.3641588Z Copyright (c) Microsoft Corporation.  All rights reserved.
    2018-10-16T00:37:05.3641723Z 
    2018-10-16T00:37:06.8873531Z Starting test execution, please wait...
    2018-10-16T00:37:51.9955035Z [xUnit.net 00:00:40.80]     SGIntegrationTests.HomeControllerShould.IndexContentTypeIsTextHtml [FAIL]
    2018-10-16T00:37:52.0883568Z Failed   SGIntegrationTests.HomeControllerShould.IndexContentTypeIsTextHtml
    2018-10-16T00:37:52.0884088Z Error Message:
    2018-10-16T00:37:52.0884378Z  Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException : Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937. Exception Message: Tried the following 3 methods to get an access token, but none of them worked.
    2018-10-16T00:37:52.0884737Z Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937. Exception Message: Tried to get token using Managed Service Identity. Access token could not be acquired. MSI ResponseCode: BadRequest, Response: {"error":"invalid_request","error_description":"Identity not found"}
    2018-10-16T00:37:52.0884899Z Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937. Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Visual Studio Token provider file not found at "C:UsersVssAdministratorAppDataLocal.IdentityServiceAzureServiceAuth	okenprovider.json"
    2018-10-16T00:37:52.0885142Z Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937. Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. Process took too long to return the token.
    2018-10-16T00:37:52.0885221Z 
    2018-10-16T00:37:52.0885284Z Stack Trace:
    2018-10-16T00:37:52.0885349Z    at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.GetAccessTokenAsyncImpl(String authority, String resource, String scope)
    2018-10-16T00:37:52.0885428Z    at Microsoft.Azure.KeyVault.KeyVaultCredential.PostAuthenticate(HttpResponseMessage response)
    2018-10-16T00:37:52.0885502Z    at Microsoft.Azure.KeyVault.KeyVaultCredential.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    2018-10-16T00:37:52.0886831Z    at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)
    2018-10-16T00:37:52.0886887Z    at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretsAsync(IKeyVaultClient operations, String vaultBaseUrl, Nullable`1 maxresults, CancellationToken cancellationToken)
    2018-10-16T00:37:52.0886935Z    at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()
    2018-10-16T00:37:52.0887000Z    at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()
    2018-10-16T00:37:52.0887045Z    at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
    2018-10-16T00:37:52.0887090Z    at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
    2018-10-16T00:37:52.0887269Z    at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
    2018-10-16T00:37:52.0887324Z    at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
    2018-10-16T00:37:52.0887371Z    at Microsoft.AspNetCore.TestHost.TestServer..ctor(IWebHostBuilder builder, IFeatureCollection featureCollection)
    2018-10-16T00:37:52.0887433Z    at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateServer(IWebHostBuilder builder)
    2018-10-16T00:37:52.0887477Z    at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()
    2018-10-16T00:37:52.0887525Z    at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)

---

Update

I have found only 1 related post to this issue: https://social.msdn.microsoft.com/Forums/en-US/0bac778a-283a-4be1-bc75-605e776adac0/managed-service-identity-issue?forum=windowsazurewebsitespreview. But the post is related to deploying an application into an azure slot. I am merely trying to build my application in a build pipeline.

I am still trying to solve this issue and am not sure what the best way to provide the required access is.

---

Update 2

I have still not found a solution for this. I am lost on how to get my pipeline to run my test without issues. I saw that the release pipeline you have the options of running tests too. But these seem to take .dll files and my build pipeline drop file only has the web app (I don't see any of the test projects published drop file). Not sure if that is even a possibility.

---

Update 3

I managed to get it to work by using the last option provided here: https://docs.microsoft.com/en-us/azure/key-vault/service-to-service-authentication#connection-string-support

I tried the other ways of using a certificate but anytime {CurrentUser} is provided in a connection string, the build pipeline fails. It works on my local machine but not in the build pipeline.

To get it to work, I had to do three things:

• Log in to Azure. Setup a new app registration in Azure AD
• In your new AD app registration, create a new client secret

• Provide your new AD App access to your key vault. Go into your key vault access policies and add the app that you created in your AD with read access to your secrets.

• Modified my call to AzureServiceTokenProvier() in my Program.cs file as follows:

   var azureServiceTokenProvider = new AzureServiceTokenProvider("connectionString={your key vault endpoint};RunAs=App;AppId={your app id that you setup in Azure AD};TenantId={your azure subscription};AppKey={your client secret key}")
  

Note that your client secret has to be formatted correctly. The app registrations (preview) generates a random secret key. Sometimes this key does not work in the connection string (throws an error as incorrectly formatted). Either try generating your own key in the non-preview version of app registration or generate a new key and try again.

After that I was able to run my integration test in my build pipeline successfully and create a release to my web app in Azure. I'm not satisfied with this approach because although it works, its exposing a secret value in the code itself. Manages service identity does not need to be turned on due to above approach. I feel that this is extremely bad in that regard.

There has to be a better way than this. One option is not to run the integration test in the build pipeline. Not sure if this is the correct approach. I'm still hoping someone will be able to provide a better approach to this or explain if my approach is okay to use.

解决方案

You should not do the integration test of authentication to Azure KeyVault within Azure DevOps Pipelines build, because you are using Azure DevOps default hosted agents.

By default, the Azure DevOps Pipelines are using basic default hosted agents, and these hosted agents are not accessible from your Azure subscription. These are not surprising, because these hosted agents are common agents for all common build needs, including build/compile, running unit tests, getting test coverages, and all of these tasks has no other additional features such as having ActiveDirectory, database, and other actual authentication/requests to other party such as authentication to any Azure Keyvault. Therefore these agents by default are not registered in your Azure subscription.

If you want to have successful integration tests for these special needs, you have to create your own agents for Azure DevOps Pipelines build and release. Therefore, there is no other way to force Azure DevOps default agent to run your KeyVault authentication tests, other than creating your own agents and configure your Azure DevOps to use your own agents.

To create your own agents, consult this documentation from Microsoft:

https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=vsts#install

UPDATE 29th October, 2018:

For more clarity, I also reply for your "Update 3" workaround. There is no guarantee that your workaround will work nicely when Microsoft updates the Azure DevOps' default hosted agent. Therefore I also need to add more point: it's not a good practice to have integration test that relies on other party beyond the realm of your Azure DevOps Pipelines build such as connecting to a database server or using external authentications (even on Azure KeyVault) within your CI, especially if you are using Microsoft's default hosted agents.

Not just it will be error-prone due to invalid authentication configuration, but there's no guarantee that the further updates on the default hosted-agents would guarantee your third-party logic test will work.

这篇关于VSTS 构建管道:测试连接到 Azure Key Vault 失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！