TestNG 重试失败的测试不会输出正确的测试结果 [英] TestNG retrying failed tests doesn't output the correct test results
问题描述
设置:我有一个扩展 IRetryAnalyzer 的类，并实现了一个简单的重试逻辑，覆盖了以下方法:公共布尔重试(ITestResult 结果){
Setup: I have a class that extends the IRetryAnalyzer and have implemented a simple retry logic overriding the following method: public boolean retry(ITestResult result) {
我有另一个类扩展类 TestListenerAdapter 重试失败的测试，直到它们通过或报告失败.我已经实现了覆盖以下方法的逻辑:public void onTestFailure(ITestResult 结果) {
I have another class that extends the class TestListenerAdapter that retries tests that failed until they pass or report failures. I have implemented my logic overriding the following method: public void onTestFailure(ITestResult result) {
场景:我总共有 10 个测试.10 次测试中有 1 次失败 2 次，并通过我的重试逻辑在第三次尝试中通过.测试结果显示如下:测试总数:12，失败:2，跳过:0
Scenario: I have a total of 10 tests. 1 out of the 10 tests fail 2 times and pass on the 3rd attempt with my retry logic. The test results show the following: Total tests: 12, Failed: 2, Skipped: 0
我想要的是输出正确数量的测试运行.并且也忽略自测试最后通过以来的 2 次失败.所以结果应该是这样的:测试总数:10，失败:0，跳过:0
What i would like is to output the correct number of tests run. And also disregard the 2 failures since the test passed at the end. So the result should look something like this: Total tests: 10, Failed:0, Skipped: 0
我在这里错过了什么?我需要修改 ITestResult 对象吗?如果是，如何?
What am i missing here? Do i need to modify the ITestResult object? If yes, how?
仅供参考:我能够使用 JUnit(实现 TestRule 接口)实现这一点.
FYI: I was able to achieve this using JUnit (implementing the TestRule interface).
提前致谢.
推荐答案
请考虑以下最大测试结果.2 次重试:
Please consider the following test results with max. 2 Retries:
- 通过 => 总体不错！
- 失败，通过 => 总体不错！
- 失败、失败、通过 => 总体不错！
- 失败，失败，失败 => 总体失败！
我所做的是创建一个 TestNg 侦听器，它扩展了默认行为并在所有测试完成后执行了一些魔术.
What i did is to create a TestNg listener which extends the default behaviour and does some magic after all tests are finished.
public class FixRetryListener extends TestListenerAdapter {
@Override
public void onFinish(ITestContext testContext) {
super.onFinish(testContext);
// List of test results which we will delete later
List<ITestResult> testsToBeRemoved = new ArrayList<>();
// collect all id's from passed test
Set <Integer> passedTestIds = new HashSet<>();
for (ITestResult passedTest : testContext.getPassedTests().getAllResults()) {
passedTestIds.add(TestUtil.getId(passedTest));
}
Set <Integer> failedTestIds = new HashSet<>();
for (ITestResult failedTest : testContext.getFailedTests().getAllResults()) {
// id = class + method + dataprovider
int failedTestId = TestUtil.getId(failedTest);
// if we saw this test as a failed test before we mark as to be deleted
// or delete this failed test if there is at least one passed version
if (failedTestIds.contains(failedTestId) || passedTestIds.contains(failedTestId)) {
testsToBeRemoved.add(failedTest);
} else {
failedTestIds.add(failedTestId);
}
}
// finally delete all tests that are marked
for (Iterator<ITestResult> iterator = testContext.getFailedTests().getAllResults().iterator(); iterator.hasNext(); ) {
ITestResult testResult = iterator.next();
if (testsToBeRemoved.contains(testResult)) {
iterator.remove();
}
}
}
}
基本上我做两件事:
- 收集所有通过的测试.如果我遇到至少一个通过测试的失败测试，我会删除失败的测试(这将涵盖上面的情况 2 和 3)
- 迭代所有失败的测试.如果我遇到以前失败的失败测试，我会删除当前失败的结果.(这实际上将涵盖情况 3 和 4).这也意味着如果有多个失败的结果，我只会保留第一个失败的结果.
为了识别测试结果，我使用以下简单的哈希函数:
To identify a testresult i use the following simple hash function:
public class TestUtil {
public static int getId(ITestResult result) {
int id = result.getTestClass().getName().hashCode();
id = 31 * id + result.getMethod().getMethodName().hashCode();
id = 31 * id + (result.getParameters() != null ? Arrays.hashCode(result.getParameters()) : 0);
return id;
}
}
这种方法也适用于数据提供者，但有一个小限制！如果您使用具有随机值的数据提供者，您会遇到问题:
This approach does work with dataproviders as well but has one small limitation! If you use dataproviders with random values you'll run into problems:
@DataProvider(name = "dataprovider")
public Object[][] getData() {
return new Object[][]{{System.currentTimeMillis()}};
}
对于失败的测试，重新评估数据提供者.因此，您将获得一个新的时间戳，同一方法的 result1 和 result2 将导致不同的哈希值.解决方案是在 getId() 方法中包含 parameterIndex 而不是参数，但 ITestResult 中似乎不包含这样的值.
For failed tests the dataprovider is reevaluated. Therefore you will get a new timestamp and the result1 and result2 for the same mehod will result in different hash values. Solution would be to include the parameterIndex in the getId() Method instead of parameters but it seems such a value is not included in ITestResult.
将此简单示例视为概念证明:
See this simple example as proof of concept:
@Listeners(value = FixRetryListener.class)
public class SimpleTest {
private int count = 0;
@DataProvider(name = "dataprovider")
public Object[][] getData() {
return new Object[][]{{"Run1"},{"Run2"}};
}
@Test(retryAnalyzer = RetryAnalyzer.class, dataProvider = "dataprovider")
public void teste(String testName) {
count++;
System.out.println("---------------------------------------");
System.out.println(testName + " " + count);
if (count % 3 != 0) {
Assert.fail();
}
count = 0;
}
}
将屈服于:
Total tests run: 2, Failures: 0, Skips: 0
这篇关于TestNG 重试失败的测试不会输出正确的测试结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
