Azure DevOps 管道:本地通过的测试在管道上失败 [英] Azure DevOps Pipeline: Tests that pass locally fail on the pipeline

问题描述

我们的团队正在尝试实施 Azure DevOps 管道，但我们遇到了一些问题.作为背景，我们在自托管的 Windows 服务器上运行测试.我们完成了所有工作的前端测试，但现在我们遇到了让后端通过的问题.

虽然大多数测试都通过得很好，但我们有一些实例没有按预期工作.尽管它们应该能够并行运行，但我将其关闭以验证这不是问题所在.我也试过在没有任何明显变化的情况下单独运行它们.

我们得到的错误只是简单的断言失败，如下所示，没有给我们太多信息.

 错误信息:Assert.AreEqual 失败.预期:<真>.实际:<假>.

但行号指向一个模拟设置，如下所示，我不明白.

mockQueryHandler.Setup(x => x.Handle(It.IsAny())).Returns(info);

目前给我们带来最多问题的测试类是测试我们的事件警报功能的测试类.似乎第一个测试会通过，但随后的测试失败.但是，我们有管道设置来重新运行任何失败的测试 3 次，直到它放弃.因此，每次重新运行时，下一次测试都会通过.我唯一能想到的是我们的汽车模拟出于某种原因给我们带来了麻烦.我们按照AutoMock - 如何使用密钥注册进行单元测试?>

我们模拟如下:

 var IAbstractEventFactoryMock = new Mock();使用 (var mock = AutoMock.GetLoose(builder =>builder.RegisterInstance(IAbstractEventFactoryMock.Object).Keyed(EventLogFactory").Keyed(ArpEventLogFactory"))) {...}

下面是我们的 .runsettings 文件

<运行配置><ResultsDirectory>.\TestResults</ResultsDirectory></运行配置><MSTest></MSTest></运行设置>

下面是我们用来运行测试的 YAML

步骤:- 任务:VSTest@2displayName: '运行所有测试'输入:testAssemblyVer2: |**\*ID_Test*.dll!**\*TestAdapter.dll!**\obj\**searchFolder: '$(System.DefaultWorkingDirectory)/ID_Test'runSettingsFile: 'ID_Test/.runsettings'并行运行:假runTestsInIsolation: 真代码覆盖启用:假testRunTitle: '单元测试'平台:'$(构建平台)'配置:'$(BuildConfiguration)'诊断启用:真rerunFailedTests: 真continueOnError: 真

同样，这些测试在本地都没有问题地通过，如果我在管道上运行它们足够多的时间，它们最终会通过.所以我不认为问题出在代码本身.似乎测试没有得到正确设置或清理.我主要是在寻找有关配置 yaml 或 runsettings 以尝试解决此问题的其他方式的任何想法.预先感谢您提供的任何帮助，如果我能提供任何其他信息以提供帮助，请告诉我.

解决方案

原来这只是我们测试的一个问题.在一些地方，我们会使用在本地很好的当前时间，但服务器的执行时间要长得多，这会产生一些错误.在执行时间与在本地运行测试相似的代理上测试管道帮助我发现管道没有问题.

Our team is trying to implement Azure DevOps pipeline but we are having some issues. For background, we are running the tests on our self-hosted windows server. We got the front-end tests to all work but now we are having issues getting the backend to pass.

While the majority of the tests are passing just fine we have a few instances where it is not working as expected. Although they should be able to run in parallel, I turned that off to verify that wasn't the issue. I've also tried running them in isolation without any noticeable change.

The errors we get are just simple assert failures like the one below that don't give us much information.

 Error Message:
   Assert.AreEqual failed. Expected:<True>. Actual:<False>.

But the line numbers point to a mock setup like the one below which I don't understand.

mockQueryHandler.Setup(x => x.Handle(It.IsAny<FindQuery>())).Returns(info);

Currently the test class that is giving us the most problems is one that tests our event alert functionality. It seems that the first test will pass but the subsequent ones fail. However, we have the pipeline setup to rerun any failed tests three times until it gives up. So on every rerun the next test will pass. The only thing I can think of is that our auto mocks are giving us trouble for some reason. We set them up in accordance with AutoMock - How to unit test with Keyed Registrations?

We mock it out as follows:

  var IAbstractEventFactoryMock = new Mock<IAbstractEventFactory>();
  using (var mock = AutoMock.GetLoose(builder => builder.RegisterInstance(IAbstractEventFactoryMock.Object).Keyed<IAbstractEventFactory>("EventLogFactory").Keyed<IAbstractEventFactory>("ArpEventLogFactory"))) {
...
}

Below is our .runsettings file

<RunSettings>  
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
  </RunConfiguration>

  <MSTest>
  </MSTest>
</RunSettings>

Below is the YAML we are using to run the tests

steps:
- task: VSTest@2
  displayName: 'Run All Tests'
  inputs:
    testAssemblyVer2: |
     **\*ID_Test*.dll
     !**\*TestAdapter.dll
     !**\obj\**
    searchFolder: '$(System.DefaultWorkingDirectory)/ID_Test'
    runSettingsFile: 'ID_Test/.runsettings'
    runInParallel: false
    runTestsInIsolation: true
    codeCoverageEnabled: false
    testRunTitle: 'Unit Tests'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: true
    rerunFailedTests: true
  continueOnError: true

Again, these tests all pass without problems locally and they pass eventually if I run them enough times on the pipeline. So I don't think the problem is in the code itself. It seems to be that the tests aren't getting setup or cleaned properly. I am primarily looking for any ideas on other ways I can configure my yaml or runsettings to try to fix this issue. Thank you in advance for any help and let me know if I can provide any additional information to help.

解决方案

Turned out this was just an issue with our tests. In a few places we would use the current time which was fine locally, but the server had a much higher execution time that created a few errors. Testing the pipeline on an agent that had a similar execution time as running the tests locally helped me discover that it wasn't an issue with the pipeline.

这篇关于Azure DevOps 管道:本地通过的测试在管道上失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！