AWS设备场似乎正在忽略TestNG批注 [英] AWS Device farm seems to be ignoring TestNG annotations

问题描述

我已成功上传并在AWS设备场上运行测试。在本地，我正在使用诸如@Test（enabled = false，dependsOnGroups = Login）之类的有趣事物来标记当时要运行的测试以及应执行的顺序。在本地，这一切都按预期正常进行。在将Maven构建的zip文件上传到设备场并执行测试运行之后，就会发生问题。

I have successfully uploaded and run my tests on AWS device farm. Locally, I'm using fun things like @Test(enabled = false, dependsOnGroups = "Login") to mark which tests to run at the time, and what order they should execute. Locally, this all works fine and dandy as expected. The problem happens after I upload the zip of the maven build to device farm and perform a test run.

查看设备场中的日志，它不在乎是否 enabled设置为true或false，无论如何都会运行。它还将忽略 group =和 dependsOnGroups注释。这非常重要，因为如果我没有先登录，所有其他测试都将失败。更糟糕的是，后续的失败测试将不会被跳过，因此AWS会从中高兴地向我收取更多的费用。

Looking at the logs from device farm, it doesn't care if "enabled" is set to true or false, it'l run things regardless. It also ignores the "group=" and "dependsOnGroups" annotations. This is super important, since all other tests will fail if I'm not logged in first. Worse, the subsequent failing tests will not be skipped, so AWS is happily charging me more money off of this.

我尝试使用@Test（priority = blah），但是它也忽略了这一点。似乎唯一值得尊重的是@BeforeSuite和@AfterSuite之类的东西。

I tried using @Test(priority=blah), but it's ignoring that too. The only thing it seems to be respecting are things like @BeforeSuite and @AfterSuite.

任何人都遇到这种情况或对为什么发生这种情况有任何想法吗？

Anyone run into this or have any ideas why this is happening?

推荐答案

我是AWS设备场上的工程师。

I am an engineer working on AWS Device Farm.

1） 已启用注释标记

我刚刚验证了您对我们的TestNG解析器的正确性，忽略了注释中的已启用标记并且始终包含测试，甚至如果已禁用。乍一看，这似乎是一个简单直接解决的问题。假设是最佳情况，我们将尝试使其尽快修复并投入生产。

I just verified that you are correct about our TestNG parser ignoring the "enabled" flag on annotations and always including the test, even if it is disabled. At first glance, this appears to be a simple issue with a straight forward fix. Assuming best case scenario, we will try and have it fixed and live in production as soon as we can.

2） dependsOnGroups注释字段

这个答案要复杂一些。到目前为止，AWS Device Farm不支持​​ dependsOnGroups 或 dependsOnMethods 注释字段。

The answer to this one is a bit more complex. As of today, AWS Device Farm does not support the dependsOnGroups or dependsOnMethods annotation fields.

有一些造成这种情况的原因，主要原因是AWS Device Farm分别执行每个 @Test 方法，每个方法都使用Appium服务器的新实例。我不会在这里深入探讨这种方法的优点/缺点，但是我想说它确实在技术和功能方面都有好处。当使用 TestNG 运行器执行单个方法时，它将仅加载单个方法的上下文，而不会加载<$中找到的指定依赖项的所有测试/套件/组。 c $ c> .jar 文件。此外，由于这些测试方法中的每一个都分别在不同的 Java 进程中执行，因此 TestNG 运行程序不会保持内存中的任何状态。这意味着它不知道它先前已经执行了测试（在另一个过程中），因此它将尝试再次运行它。

There are a few reasons for this, with the main reason being that AWS Device Farm executes each @Test method individually, each one using a fresh instance of the Appium server. There are pros/cons to this approach that I won't delve into here, but I will say that it does have benefits on both a technical and feature level. When executing individual methods with the TestNG runner, it will only load the context of an individual method, and not all tests/suites/groups of the specified dependencies found within the .jar file. Additionally, since each one of these test methods is executed in a different Java process each time, the TestNG runner does not maintain any "state" in memory. This means that it does not know that it has already executed a test previously (in a different process) so it will attempt to run it again.

3）执行测试的组

我们目前不向用户公开个组/排除组允许他们选择要执行或跳过的特定测试集合。但是，应该可以通过在根目录下的 testng.xml 文件中配置< groups> 条目来实现在测试包档案中上传的 *-tests.jar 文件。在这种情况下，我们的解析器应仅发现在这些组中定义的测试，而不是所有带有 @Test 注释的方法。

We currently do not expose groups/excludegroups to the user to allow them to pick specific collections of tests to execute or skip. However, this should be possible by configuring your <groups> entries in a testng.xml file placed in the root of your *-tests.jar file that is uploaded in your test package archive. In this case, our parser should only "discover" tests defined within those groups and not all @Test annotated methods. I have not tried this specific case however, so your YMMV.

希望有帮助！如果您还有其他疑问或想让我们查看特定的运行/测试包，请随时与我们联系或将URL粘贴到以前执行的运行中。

Hope that helps! If you have any additional questions or have a specific run/test package you'd like us to look at, feel free to reach out or paste a URL to a previously executed run.

这篇关于AWS设备场似乎正在忽略TestNG批注的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！