由于Azure DevOps超时，导致随机Selenium E2e测试失败，但可以在本地和远程Selenium上运行(BrowserStack Automate) [英] Random Selenium E2e Tests Fail because of timeouts on Azure DevOps but work locally and with remote Selenium (BrowserStack Automate)

问题描述

我有一套Selenium测试，可以在我的本地环境中使用Browserstack Automate完美地运行，但是在Azure DevOps上失败.

I've got a suite of Selenium tests that work perfectly in my local environment and using Browserstack Automate, but fail on Azure DevOps.

在Azure Devops上运行时，没有配置或设置更改.

There are no configuration or setting changes when running on Azure Devops.

我们在这里遵循了所有文档: https://docs.microsoft.com/zh-CN/azure/devops/pipelines/test/continuous-test-selenium?view=vsts

We've followed all the documentation here: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/continuous-test-selenium?view=vsts

随机测试失败，从来没有一样.

Random tests fail, never the same ones.

由于超时，测试总是失败.我等待页面加载5分钟，因此不会出现超时过低的情况.

The tests always fail because of timeouts. I wait for the pages to load for 5 minutes so it's not a case of the timeouts being too low.

没有防火墙，该应用程序是公共的.

There are no firewalls in place, the application is public.

身份验证总是成功的，因此测试能够加载应用程序.

Authentication always succeeds so the tests are able to load the application.

不确定下一步该怎么做.

Not sure what to try next.

下面是Azure DevOps日志的副本. 4个测试通过，但其他所有测试均失败.通常，只有4-5次测试会失败.

Below is a copy of the Azure DevOps log. 4 tests passed but all the other's failed. Usually, only 4-5 tests fail.

此测试可以使用BrowserStack Automate(远程硒)和本地方法完美运行.

This tests works perfectly using BrowserStack Automate (remote selenium) and locally.

2018-11-17T05:40:28.6300135Z  Failed   StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending
2018-11-17T05:40:28.6300461Z Error Message:
2018-11-17T05:40:28.6304198Z  Test method CS.Portal.E2e.Tests.Admin.StripeAdmin.StripeAdminTests.StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending threw exception: 
2018-11-17T05:40:28.6305677Z OpenQA.Selenium.WebDriverTimeoutException: Timed out after 300 seconds
2018-11-17T05:40:28.6307041Z Stack Trace:
2018-11-17T05:40:28.6307166Z     at OpenQA.Selenium.Support.UI.DefaultWait`1.ThrowTimeoutException(String exceptionMessage, Exception lastException)
2018-11-17T05:40:28.6307999Z    at OpenQA.Selenium.Support.UI.DefaultWait`1.Until[TResult](Func`2 condition)
2018-11-17T05:40:28.6308188Z    at CS.Portal.E2e.Tests.Utility.WebDriverUtilities.WaitForElement(IWebDriver driver, By by, Boolean mustBeDisplayed) in D:\a\1\s\CS.Portal.E2e.Tests\Utility\WebDriverUtilities.cs:line 26
2018-11-17T05:40:28.6319651Z    at CS.Portal.E2e.Tests.Admin.StripeAdmin.StripeAdminTests.StripeAdmin_WhenOnTab_DefaultSortIsByIdDescending() in D:\a\1\s\CS.Portal.E2e.Tests\Admin\StripeAdmin\StripeAdminTests.cs:line 51
2018-11-17T05:40:28.6319982Z 
2018-11-17T05:40:34.4671568Z Results File: D:\a\1\s\TestResults\VssAdministrator_factoryvm-az416_2018-11-17_03_08_24.trx
2018-11-17T05:40:34.4692222Z 
2018-11-17T05:40:34.4695222Z Attachments:
2018-11-17T05:40:34.4697610Z   D:\a\1\s\TestResults\672f4d28-5082-42e9-a7e7-f5645aadcfd8\VssAdministrator_factoryvm-az416 2018-11-17 03_02_43.coverage
2018-11-17T05:40:34.4697943Z 
2018-11-17T05:40:34.4698278Z Total tests: 34. Passed: 4. Failed: 30. Skipped: 0.

推荐答案

以下是我要做的一些步骤:

Here are some steps I would do :

1. 在类似情况下，对我们有帮助的是暂时将录像机添加到测试中，然后从头到尾观看VM上的测试执行过程.视频中可能有一些线索，可以帮助您了解实际出了什么问题我可以找到有关C#示例的链接

此外，我还要仔细检查以确保Azure上的浏览器版本与运行良好的浏览器版本完全相同.使它们相同对于确保没有魔术"至关重要.与默认浏览器窗口大小相同.

Also, I would double check to make sure browser versions on Azure are exactly the same as in the run where it all works well. Making them the same is crucial to make sure there is no 'magic'. Same for the default browser window size.

我将对不同测试失败的地方进行更详细的分析.

I would do more detailed analysis of places where different tests fail.

有可能发现不同测试失败之间的相似之处.点击后是否总是会发生?重新加载页面后?还有其他类似的东西吗?如果是，请尝试使用最怪异但又简单又有时可以挽救生命的解决方案，并在故障之前的操作之前/之后增加3-5秒的睡眠时间. (添加有条件的睡眠仅在Azzure运行时才发生)(是的，不建议睡眠，{很多不建议为什么睡眠的众所周知的信息可能在这里}，但是). ..如果它们神奇地保存了您的跑步，则可以通过一些智能等待来确保替换它们)
• 是否有可能在某个特定时间发生故障?运行后经过同一时间开始吗?白天在同一时间?

• is it possible to spot the similarities between different test failures. Does it always happen after clicks? after reloading pages? after anything else that is similar? If yes - try with the weirdest yet simple and sometimes lifesaving solution and add a 3-5 sec sleep before/after an action that is before the failure. (add sleeps with a condition to only happen when it's an Azzure run) (yes, sleeps are not recommended and {a lot of well-known info why they are not recommended could have been here} but... if they magically save your runs you can then replace them with some smart waits for sure)
• is it possible that failures happen at some certain time? After same time after run start? At the same time during the day?

如果在代码中使用日期/时间API，请确保系统时间/语言环境/时区设置完全相同.否则，测试运行期间的日子不会改变.总而言之-围绕日期进行调查.

If you use Date/time APIs in your code, make sure the System time/locale/timezone settings are exactly the same. Or that the days do not change during the test runs. All in all - investigate around dates.

我知道以上内容更像是一个一般性建议，但是从我的经验来看，这种随机失败"可能是由任何看起来不值得关注"的东西引起的.

I know that the above is more like a general advice, but from my experience such "random failures" could be caused by literally anything that seems "not worth attention".

这篇关于由于Azure DevOps超时，导致随机Selenium E2e测试失败，但可以在本地和远程Selenium上运行(BrowserStack Automate)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！