由于 Azure DevOps 超时，随机 Selenium E2e 测试失败，但在本地和远程 Selenium 上工作(BrowserStack Automate) [英] Random Selenium E2e Tests Fail because of timeouts on Azure DevOps but work locally and with remote Selenium (BrowserStack Automate)

问题描述

我有一套 Selenium 测试，可以在我的本地环境中完美运行并使用 Browserstack Automate，但在 Azure DevOps 上失败.

I've got a suite of Selenium tests that work perfectly in my local environment and using Browserstack Automate, but fail on Azure DevOps.

在 Azure Devops 上运行时没有配置或设置更改.

There are no configuration or setting changes when running on Azure Devops.

我们遵循了此处的所有文档:https://docs.microsoft.com/en-us/azure/devops/pipelines/test/continuous-test-selenium?view=vsts

We've followed all the documentation here: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/continuous-test-selenium?view=vsts

随机测试失败，永远不会相同.

Random tests fail, never the same ones.

由于超时，测试总是失败.我等待页面加载 5 分钟，所以这不是超时太低的情况.

The tests always fail because of timeouts. I wait for the pages to load for 5 minutes so it's not a case of the timeouts being too low.

没有防火墙，应用程序是公开的.

There are no firewalls in place, the application is public.

身份验证总是成功，因此测试能够加载应用程序.

Authentication always succeeds so the tests are able to load the application.

不确定接下来要尝试什么.

Not sure what to try next.

以下是 Azure DevOps 日志的副本.4 项测试通过了，但其他所有测试都失败了.通常，只有 4-5 个测试失败.

Below is a copy of the Azure DevOps log. 4 tests passed but all the other's failed. Usually, only 4-5 tests fail.

此测试使用 BrowserStack Automate(远程 selenium)和本地完美运行.

This tests works perfectly using BrowserStack Automate (remote selenium) and locally.

2018-11-17T05:40:28.6300135Z  Failed   StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending
2018-11-17T05:40:28.6300461Z Error Message:
2018-11-17T05:40:28.6304198Z  Test method CS.Portal.E2e.Tests.Admin.StripeAdmin.StripeAdminTests.StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending threw exception: 
2018-11-17T05:40:28.6305677Z OpenQA.Selenium.WebDriverTimeoutException: Timed out after 300 seconds
2018-11-17T05:40:28.6307041Z Stack Trace:
2018-11-17T05:40:28.6307166Z     at OpenQA.Selenium.Support.UI.DefaultWait`1.ThrowTimeoutException(String exceptionMessage, Exception lastException)
2018-11-17T05:40:28.6307999Z    at OpenQA.Selenium.Support.UI.DefaultWait`1.Until[TResult](Func`2 condition)
2018-11-17T05:40:28.6308188Z    at CS.Portal.E2e.Tests.Utility.WebDriverUtilities.WaitForElement(IWebDriver driver, By by, Boolean mustBeDisplayed) in D:a1sCS.Portal.E2e.TestsUtilityWebDriverUtilities.cs:line 26
2018-11-17T05:40:28.6319651Z    at CS.Portal.E2e.Tests.Admin.StripeAdmin.StripeAdminTests.StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending() in D:a1sCS.Portal.E2e.TestsAdminStripeAdminStripeAdminTests.cs:line 51
2018-11-17T05:40:28.6319982Z 
2018-11-17T05:40:34.4671568Z Results File: D:a1sTestResultsVssAdministrator_factoryvm-az416_2018-11-17_03_08_24.trx
2018-11-17T05:40:34.4692222Z 
2018-11-17T05:40:34.4695222Z Attachments:
2018-11-17T05:40:34.4697610Z   D:a1sTestResults672f4d28-5082-42e9-a7e7-f5645aadcfd8VssAdministrator_factoryvm-az416 2018-11-17 03_02_43.coverage
2018-11-17T05:40:34.4697943Z 
2018-11-17T05:40:34.4698278Z Total tests: 34. Passed: 4. Failed: 30. Skipped: 0.

推荐答案

这里有一些我会做的步骤:

Here are some steps I would do :

1. 在类似情况下帮助我们的是临时添加一个录像机进行测试，然后在虚拟机上观看从开始到失败的测试执行过程.视频中可能有一些线索可以帮助您了解实际出了什么问题 我能够找到 C# 示例的链接

此外，我会仔细检查以确保 Azure 上的浏览​​器版本与运行良好的运行版本完全相同.使它们相同对于确保没有魔法"至关重要.默认浏览器窗口大小相同.

Also, I would double check to make sure browser versions on Azure are exactly the same as in the run where it all works well. Making them the same is crucial to make sure there is no 'magic'. Same for the default browser window size.

我会对不同测试失败的地方做更详细的分析.

I would do more detailed analysis of places where different tests fail.

• 是否有可能发现不同测试失败之间的相似性.它总是在点击后发生吗?重新加载页面后?在其他类似的事情之后?如果是 - 尝试使用最奇怪但有时可以挽救生命的解决方案，并在失败之​​前的操作之前/之后添加 3-5 秒的睡眠.(将睡眠添加到仅在 Azzure 运行时才发生的条件)(是的，不推荐睡眠，{很多众所周知的信息为什么不推荐它们}但是...如果他们神奇地保存了您的运行，那么您肯定可以用一些智能等待替换它们)
• 是否有可能在某个特定时间发生故障?在运行开始后的同一时间之后?白天的同一时间?

• is it possible to spot the similarities between different test failures. Does it always happen after clicks? after reloading pages? after anything else that is similar? If yes - try with the weirdest yet simple and sometimes lifesaving solution and add a 3-5 sec sleep before/after an action that is before the failure. (add sleeps with a condition to only happen when it's an Azzure run) (yes, sleeps are not recommended and {a lot of well-known info why they are not recommended could have been here} but... if they magically save your runs you can then replace them with some smart waits for sure)
• is it possible that failures happen at some certain time? After same time after run start? At the same time during the day?

如果您在代码中使用日期/时间 API，请确保系统时间/区域设置/时区设置完全相同.或者在测试运行期间日期不会改变.总而言之 - 围绕日期进行调查.

If you use Date/time APIs in your code, make sure the System time/locale/timezone settings are exactly the same. Or that the days do not change during the test runs. All in all - investigate around dates.

我知道以上更像是一般性建议，但根据我的经验，这种随机故障"可能是由任何看起来不值得关注"的东西引起的.

I know that the above is more like a general advice, but from my experience such "random failures" could be caused by literally anything that seems "not worth attention".

这篇关于由于 Azure DevOps 超时，随机 Selenium E2e 测试失败，但在本地和远程 Selenium 上工作(BrowserStack Automate)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！