cucumber.json 报告被重新运行方案报告覆盖 [英] cucumber.json report getting overwritten by rerun scenario report
问题描述
我有 UI 测试项目和一个具有相同技术堆栈(JAVA1.8、Cucumber-JVM、JUnit、Maven)的 API 测试项目，这两个项目都向我展示了这个问题.可能是因为两者都存在相同的依赖集.
I have got UI Test project and a API test project with same technology stack (JAVA1.8, Cucumber-JVM, JUnit, Maven) and both projects are showing me this problem. Probably because same set of dependencies are present in both.
我采用了 Flaky 测试重新运行机制，使用 maven-surefire-plugin 内置功能 <rerunFailingTestsCount>1</rerunFailingTestsCount>.另外，我添加了基于 <groupId>io.cucumber</groupId> 而不是 <groupId>info.cukes</groupId> 的黄瓜依赖项.这两个都有自己版本的 cucumber-java 和 cucumber-jvm 依赖项.
I have employed the Flaky test re-run mechanism using maven-surefire-plugin build-in functionality <rerunFailingTestsCount>1</rerunFailingTestsCount>. Also, I have cucumber dependencies added based on <groupId>io.cucumber</groupId> and not <groupId>info.cukes</groupId>. Both these have their own version of cucumber-java and cucumber-jvm dependencies.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
<configuration>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
</configuration>
<executions>
<execution>
<id>acceptance-test</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<forkCount>1</forkCount>
<reuseForks>true</reuseForks>
<includes>
<include>**/*Runner.class</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-spring</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-jvm</artifactId>
<version>2.4.0</version>
<type>pom</type>
</dependency>
仅运行器文件代码
@RunWith(Cucumber.class)
@ContextConfiguration(locations = {"file:/src/test/resources/spring-config.xml"})
@CucumberOptions(
glue = "com.test.uitest",
features = "classpath:cucumber",
tags = {"~@ignore","@ui_home"},
monochrome = true,
plugin = {"pretty", "html:target/cucumber-reports",
"json:target/cucumber-reports/cucumber.json",
"rerun:target/rerun.txt"} //Creates a text file with failed scenarios
)
public class AllTestsRunner {
}
现在显然，我需要另一个运行器，其中包含以下代码(根据 StackOverflow 上的其他论坛和线程)
Now Aparently, I need to have another runner with following code in it (as per other forums and threads here on StackOverflow)
@RunWith(Cucumber.class)
@CucumberOptions(
monochrome = true,
glue = "com.test.uitest",
features = "@target/rerun.txt", //Cucumber picks the failed scenarios from this file
format = {"pretty", "html:target/rerun-reports",
"json:target/cucumber-reports/rerun-report.json"}
)
public class FailedTestsRunner {
}
但我不需要这个第二个跑步者，因为只有一个跑步者在上面，重新运行机制绝对出色.甚至，不需要在 1st runner 中生成 rerun.txt 文件.maven-surefire 插件(v_2.21.0)中的内置机制以及 io.cucumber v_2.4.0 可以完美运行，如果在第一次执行期间出现任何场景失败，它会自动重新运行，而不会将其记录在 rerun.txt 文件中.
But I don't need to have this 2nd runner, as the rerun mechanism works absolutely brilliantly with just the one runner on top. Even, there is no need of rerun.txt file generated in 1st runner.The in-build mechanism within maven-surefire plugin (v_2.21.0) along with io.cucumber v_2.4.0 works perfectly and if any scenario fails during 1st execution, it reruns automatically without recording it inside rerun.txt file.
我的功能文件中有 5 个场景.如果他们都在第一次运行中通过.它成功生成了带有 cucumber.json 报告的报告，显示了所有 5 个场景.但是，如果(比如说)5 个场景中有 2 个失败，并且这些场景在重新运行机制中自动执行，那么 cucumber.json 报告文件只记录这两个场景的结果，而不是所有 5 个场景的结果.如果这 2 个场景在重新运行中通过，则整体构建通过；如果这 2 个场景失败，则整体构建失败.这是正确的，但我的问题是 cucumber.json 被重新运行机制覆盖.我曾尝试使用 maven-cucumber-reporting 插件 v_3.16.0 但它实际上读取 cucumber.json 文件本身，因此不能解决我的问题.任何帮助，将不胜感激.
I have 5 scenarios in my feature file. If all of them pass in 1st run. It successfully generates the report with cucumber.json report showing all 5 scenarios.
But, if (say) 2 out of 5 scenarios fails, and those gets executed automatically in a rerun mechanism, the cucumber.json report file only recording results of those two scenarios and not the all 5 scenarios. Overall build PASSES if those 2 scenarios passes in rerun or FAILs if those 2 scenarios fails. That's correct, but my problem is with the cucumber.json getting overwritten by rerun mechanism.
I have tried to use the maven-cucumber-reporting plugin v_3.16.0 but it actually reads the cucumber.json file itself and hence don't resolve my problem. Any help would be appreciated.
推荐答案
好消息是你没有做错任何事！
The good news is you're not doing anything wrong!
坏消息是您观察到的结果完全符合预期.这是链接不同工具(Surefire --> JUnit --> Cucumber)的自然结果，否则这些工具彼此不知道.从 Cucumber 的角度来看，重新运行似乎是一个全新的执行，因此它会很高兴地覆盖旧报告.只有在链的开始，才有可能创建准确的报告.
The bad news is that the results you're observing are entirely as expected. This is a natural consequence of chaining different tools (Surefire --> JUnit --> Cucumber) that are otherwise unaware of each other. From the perspective of Cucumber it would appear that the rerun is an entirely new execution so it will happily overwrite the old reports. Only at the start of the chain it is possible to create accurate reports.
因此，您的选择从最少到最多以及从最差到最好的质量是:
As such your options from least to most effort and worst to best quality are:
使用 Surefire 生成的报告.
Use the reports generated by Surefire.
编写您自己的插件来附加结果而不是覆盖它们.
Write your own plugin that appends results rather then overwriting them.
删除了使用 cucumber-jvm-parallel-plugin 的建议 为每个场景创建单独的单元测试.这实际上是行不通的.
Removed the suggestion to use the cucumber-jvm-parallel-plugin create a individual unit test for each scenario. This won't actually work.
这篇关于cucumber.json 报告被重新运行方案报告覆盖的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
