如果在 Docker 下运行的 .NET Core 单元测试的代码覆盖率低于 90%，则中断 TeamCity 中的构建 [英] Breaking the build in TeamCity if .NET Core unit tests running under Docker have code-coverage less than 90%

问题描述

我最近一直在研究 Docker，以及如何使用 TeamCity 在 Docker 容器中运行 .NET Core 单元测试作为构建管道的一部分.我将此添加为 Dockerfile 中的最后一行，以便能够运行测试:

I have recently been looking at Docker, and how I can use TeamCity to run .NET Core unit tests in Docker containers as part of my build pipe-line. I add this as the final line in my Dockerfile to be able to run tests:

ENTRYPOINT ["dotnet", "test", "--verbosity=normal"]

然后在 TeamCity 使用命令行中的 docker-compose 构建和运行的撰写文件中引用这些 Dockerfile.

These Dockerfiles are then referenced in compose files which TeamCity builds and runs using docker-compose in the command line.

我现在成功地完成了这项工作.下一个挑战是在单元/集成测试覆盖率低于 90% 或其他值时中断构建 - 请不要就此争论！

I have this working successfully now. Next challenge is to break the build if unit/integration test coverage is less than 90% - or some other value - no arguments about this please!

我成功地使用了 coverlet.msbuild NuGet 依赖项来测量代码覆盖率，作为我构建的一部分.这在 TeamCity 中也能正常工作，而且我在 TeamCity 构建中看到了输出.

I'm successfully using the coverlet.msbuild NuGet dependency to measure code-coverage as part of my build. This works fine in TeamCity too, and I see the output in my TeamCity build.

我通过将 coverlet.msbuild 添加到我的每个测试项目并将 Dockerfile 入口点更改为:

I got this working by adding coverlet.msbuild to each of my test projects, and changing the Dockerfile entry-point to:

ENTRYPOINT ["dotnet", "test", "--verbosity=normal", "/p:CollectCoverage=true", "/p:Threshold=90", "/p:ThresholdType=line"]

TeamCity 构建输出显示带有结果的 ASCII 表，但到目前为止，如果代码覆盖率不够高，我还没有找到打破构建的好方法.如果代码覆盖率太低，TeamCity 不会将构建标记为失败，如果足够公平，因为它不是通灵的！

The TeamCity build output shows the ASCII tables with the results in, but as of yet I've not been able to find a good way to break the build if the code-coverage isn't high enough. Left to its own devices, TeamCity doesn't mark builds as failing if the code-coverage is too low, which if fair enough as it's not psychic!

我天真地认为我可以在 TeamCity 中创建一个失败条件，它会检测以下文本的存在:

I naively thought I could create a failure condition in TeamCity which would detect for the presence of the following text:

'[Assemnbly]' has a line coverage '9.8%' below specified threshold '95%'

...使用这样的正则表达式:

...using a regular expression like this:

has a line coverage '((d+(.d*)?)|(.d+))%' below specified threshold '((d+(.d*)?)|(.d+))%'

然而，当被测试的 DLL 引用其他单独测试的 DLL 时，它会变得棘手，因为 coverlet.msbuild 报告所有接触"的 DLL 的覆盖率指标.例如，我有一个名为 Steve.Core.Files.Tests 的测试项目，它测试 Steve.Core.Files.但是，Steve.Core.Files 又引用了 Steve.Core.Extensions.我在自己的测试 DLL 中单独测试 Steve.Core.Extensions，因此在测试文件时我不关心该 DLL 的结果.TeamCity 中的输出如下所示:

However, when DLLs being tested reference other DLLs that are tested separately, it gets tricky because coverlet.msbuild reports coverage metrics for all "touched" DLLs. For instance, I have a test project called Steve.Core.Files.Tests which tests Steve.Core.Files. However, Steve.Core.Files in turn references Steve.Core.Extensions. I test Steve.Core.Extensions separately in its own test DLL so I don't care about the results for that DLL when testing files. The output in TeamCity looks like this:

+-----------------------+--------+--------+--------+
| Module                | Line   | Branch | Method |
+-----------------------+--------+--------+--------+
| Steve.Core.Extensions | 23.5%  | 40%    | 40%    |
+-----------------------+--------+--------+--------+
| Steve.Core.Files      | 100%   | 100%   | 100%   |
+-----------------------+--------+--------+--------+

...所以它基于 23.5% 位失败，即使有问题的 DLL 是 100%.这实际上使得检查使用 Regex 失败条件变得非常困难.

...so it fails based on the 23.5% bit, even though the DLL in question is 100%. This actually makes it very difficult to check the using a Regex failure condition.

为了使事情进一步复杂化，我使用单个动态 Dockerfile 在所有程序集中运行所有测试，原因有两个:

To complicate things further I'm running all tests in all assemblies using a single dynamic Dockerfile, for two reasons:

1. 我不想每次添加更多项目和测试时都必须更改 Dockerfile 和 docker-compose 文件(和 TeamCity).

1. I don't want to have to change the Dockerfile and docker-compose file (and TeamCity) each time I add more projects and tests.

DLL 之间存在许多依赖关系，因此构建一次并一起测试它们是有意义的.

There are many dependencies between the DLLs so it makes sense to build them once and test them all together.

这意味着我不愿意拆分测试，以便每个测试都有自己的 Dockerfile - 我知道这将允许我使用 Exclude/Include 标志来获得所需的行为.

This means I'm loath to split the tests up so that each has its own Dockerfile - I know that this would allow me to use the Exclude/Include flags to get the desired behaviour.

有人对我如何解决这个问题有任何其他想法吗?

Does anyone have any other ideas how I can solve this please?

我希望我可以在每个测试项目的级别添加一个文件，告诉它要覆盖哪些 DLL - 这将是最好的解决方案.否则，由于我在项目和测试项目之间使用严格的命名约定，我是否可以向 dotnet test 命令添加一个开关，以仅测试与测试程序集同名的程序集减去 .最后测试一下?

I'm hoping I can add a file at the level of each test project to tell it which DLLs to do coverage for - that would be the best solution. Failing that, as I use a strict naming convention between projects and test projects, can I add a switch to my dotnet test command to test only the assembly that has the same name as the test assembly minus the .Tests bit on the end?

提前致谢；感谢帮助！

干杯，

史蒂夫.

2018 年 9 月 7 日更新:

因此，我的 Dockerfile 现在特定于每个单元测试项目.它们看起来像这样并且存在于测试项目文件旁边:

So, my Dockerfiles are now specific to each unit test project. They look like this and exist in next to the test project files:

FROM microsoft/dotnet:2-sdk

# Set the working directory:
WORKDIR /src

# Copy the solution file and the NuGet.config across to the src directory:
COPY *.sln NuGet.config ./

# Copy the main source project files to the root level:
COPY */*.csproj ./

# Make directories for each project file and move the project file to the correct place:
RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done

# Restore dependencies:
RUN dotnet restore

# Copy all files so that we have all everything ready to compile:
COPY . .

# Set the flag to tell TeamCity that these are unit tests:
ENV TEAMCITY_PROJECT_NAME = ${TEAMCITY_PROJECT_NAME}

# Run the tests:
ENTRYPOINT ["dotnet", "test", "Steve.Core.Configuration.Tests/Steve.Core.Configuration.Tests.csproj", "--verbosity=normal", "/p:CollectCoverage=true", "/p:Threshold=95", "/p:ThresholdType=line", "/p:Exclude="[Steve.Core.Testing]*""]

注意排除开关，它应该停止 Steve.Core.Testing DLL 的覆盖结果，它包含在 Steve.Core.Configuration 的结果中，这是测试的主要依赖项，并且正在单元测试的项目.

Note the exclude switch which is supposed to stop the coverage results for the Steve.Core.Testing DLL being included in the results for Steve.Core.Configuration, which is the main dependency of the tests, and the project being unit tested.

我的撰写文件看起来像这样并且存在于解决方案文件旁边:

My compose file looks like this and exists next to the solution file:

version: '3.6'

services:
  # Dependencies:
  steve.core.ldap.tests.ldap:
    image: osixia/openldap
    container_name: steve.core.ldap.tests.ldap
    environment:
      LDAP_ORGANISATION: Steve
      LDAP_DOMAIN: steve.com
      LDAP_ADMIN_PASSWORD: Password1
  steve.core.data.mysql.tests.database:
    image: mysql
    container_name: steve.core.data.mysql.tests.database
    command: mysqld --default-authentication-plugin=mysql_native_password
    environment:
      - MYSQL_ROOT_PASSWORD=Password1
      - MYSQL_DATABASE=testdb
  steve.core.data.sqlserver.tests.database:
    image: microsoft/mssql-server-linux
    container_name: steve.core.data.sqlserver.tests.database
    environment:
      - MSSQL_SA_PASSWORD=Password1
      - ACCEPT_EULA=Y
      - MSSQL_PID=Developer  
  steve.core.email.tests.smtp:
    image: mailhog/mailhog 
    container_name: steve.core.email.tests.smtp  

  # Steve.Core.Configuration:
  steve.core.configuration.tests:
    image: steve.core.configuration.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Configuration.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Data.MySql:
  steve.core.data.mysql.tests:
    image: steve.core.data.mysql.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Data.MySql.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Data.SqlServer:
  steve.core.data.sqlserver.tests:
    image: steve.core.data.sqlserver.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Data.SqlServer.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Data:
  steve.core.data.tests:
    image: steve.core.data.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Data.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Email:
  steve.core.email.tests:
    image: steve.core.email.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Email.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Encryption:
  steve.core.encryption.tests:
   image: steve.core.encryption.tests:tests
   build:
     context: .
     dockerfile: Steve.Core.Encryption.Tests/Dockerfile
   environment:
     - TEAMCITY_PROJECT_NAME

  # Steve.Core.Execution:
  steve.core.execution.tests:
    image: steve.core.execution.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Execution.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Extensions:
  steve.core.extensions.tests:
    image: steve.core.extensions.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Extensions.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Files:
  steve.core.files.tests:
    image: steve.core.files.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Files.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Ldap:
  steve.core.ldap.tests:
    image: steve.core.ldap.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Ldap.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Maths:
  steve.core.maths.tests:
    image: steve.core.maths.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Maths.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Time:
  steve.core.time.tests:
    image: steve.core.time.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Time.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

当它在 TeamCity 中运行时，它只报告来自两个项目的 7 个测试(出于某种奇怪的原因)，尽管 12 个项目中有 236 个测试.

When it runs in TeamCity, it reports only 7 tests from two projects (for some strange reason) even though there are 236 tests in 12 projects.

如果有帮助，我很乐意通过电子邮件发送 TeamCity 构建的输出.

I'd be happy to email the output from the TeamCity build if it will help.

有谁知道我怎样才能让我的所有测试再次运行?

Does anyone know how I can get my tests all running again please?

谢谢，

史蒂夫.

推荐答案

因此，唯一的解决方案是将每个单元测试项目拆分为自己的组合文件，其中包含仅该测试 DLL 所需的依赖项.(例如，用于测试电子邮件 DLL 的 mailhog、用于测试数据库 DLL 的 SQL Server 等......).TeamCity 然后使用这样的单个脚本单独运行它们:

So, the only solution was to split each unit test project up into its own compose file which includes the dependencies needed for just that test DLL. (e.g. mailhog for testing email DLLs, SQL Server for testing database DLLs, etc...). TeamCity then runs them all individually using a single script like this:

docker-compose -f docker-compose-configuration-tests.yml up --force-recreate --abort-on-container-exit --build
docker-compose -f docker-compose-configuration-tests.yml down --volumes --remove-orphans

docker-compose -f docker-compose-data-mysql-tests.yml up --force-recreate --abort-on-container-exit --build
docker-compose -f docker-compose-data-mysql-tests.yml down --volumes --remove-orphans

...

每个都有自己的 Dockerfile，它构建测试 DLL 并为单元测试覆盖设置 DLL 异常.TeamCity 在单个构建步骤中吐出所有测试的结果，然后我上面问题中提到的正则表达式代码覆盖率失败条件会正确检测未达到 x% 覆盖率的测试项目并破坏构建.

Each one has its own Dockerfile which builds the test DLL and sets the DLL exceptions for unit test coverage. TeamCity spits out the results of all tests in a single build step, and the regex code coverage failure condition mentioned in my question above then correctly detects test projects that don't achieve x% coverage and breaks the build.

现在研究如何将代码检查(例如 FxCop 和 StyleCop 的现代等价物)集成到我的构建过程中...

Now to work out how to integrate code checking (e.g. the modern equivalents of FxCop and StyleCop) into my build process...

这篇关于如果在 Docker 下运行的 .NET Core 单元测试的代码覆盖率低于 90%，则中断 TeamCity 中的构建的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！