JUnit5:如何重复失败的测试? [英] JUnit5: How to repeat failed test?
问题描述
许多公司遵循的一种做法是重复进行不稳定的测试，直到通过 x 次(连续或总计).如果它执行了 n 次，但未通过至少 x 次，则将其标记为失败.
One of the practice many companies follow is to repeat unstable test until is passes x times (in a row or in total). If it is executed n times and fail to pass at least x times it is marked as failed.
TestNG支持以下注释:
TestNG supports that with the following annotation:
@Test(invocationCount = 5, successPercentage = 40)
如何使用JUnit5实现类似的功能?
How do I realize similar functionality with JUnit5?
JUnit5中有一个类似的注释，称为@RepeatedTest(5)，但它不是有条件执行的.
There's similar annotation in JUnit5, called @RepeatedTest(5) but it is not executed conditionally.
推荐答案
Ok, I took a little bit of time to whip together a little example of how to do this using the TestTemplateInvocationContextProvider, ExecutionCondition, and TestExecutionExceptionHandler extension points.
我能够处理失败的测试的方法是将它们标记为中止"，而不是让它们断然失败(以便整个测试执行不会将其视为失败)，并且仅在我们无法完成时才通过测试获得最少的成功运行次数.如果最小数量的测试已经成功，则我们还将其余测试标记为已禁用".在
The way I was able to handle failing tests was to mark them as "aborted" rather than let them flat out fail (so that the entire test execution does not consider it a failure) and only fail tests when we can't get the minimum amount of successful runs. If the minimum amount of tests has already succeeded, then we also mark the remaining tests as "disabled". The test failures are tracked in a ExtensionContext.Store so that the state can be looked up at each place.
这是一个非常粗糙的示例，肯定有一些问题，但是希望可以作为如何编写不同注释的示例.我最终用Kotlin编写了它:
This is a very rough example that definitely has a few problems but can hopefully serve as an example of how to compose different annotations. I ended up writing it in Kotlin:
@Retry -esque注释基于TestNG示例:
@Retry-esque annotation loosely based on the TestNG example:
import org.junit.jupiter.api.TestTemplate
import org.junit.jupiter.api.extension.ExtendWith
@TestTemplate
@Target(AnnotationTarget.FUNCTION)
@ExtendWith(RetryTestExtension::class)
annotation class Retry(val invocationCount: Int, val minSuccess: Int)
TestTemplateInvocationContext 用于模板化测试:
TestTemplateInvocationContext used by templatized tests:
import org.junit.jupiter.api.extension.Extension
import org.junit.jupiter.api.extension.TestTemplateInvocationContext
class RetryTemplateContext(
private val invocation: Int,
private val maxInvocations: Int,
private val minSuccess: Int
) : TestTemplateInvocationContext {
override fun getDisplayName(invocationIndex: Int): String {
return "Invocation number $invocationIndex (requires $minSuccess success)"
}
override fun getAdditionalExtensions(): MutableList<Extension> {
return mutableListOf(
RetryingTestExecutionExtension(invocation, maxInvocations, minSuccess)
)
}
}
Simple 实现 然后，测试使用者: IntelliJ中的测试输出如下: And the test output in IntelliJ looks like: 我不知道从 I don't know if throwing a 这篇关于JUnit5:如何重复失败的测试?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！TestTemplateInvocationContextProvider扩展名:import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.ExtensionContextException
import org.junit.jupiter.api.extension.TestTemplateInvocationContext
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider
import org.junit.platform.commons.support.AnnotationSupport
import java.util.stream.IntStream
import java.util.stream.Stream
class RetryTestExtension : TestTemplateInvocationContextProvider {
override fun supportsTestTemplate(context: ExtensionContext): Boolean {
return context.testMethod.map { it.isAnnotationPresent(Retry::class.java) }.orElse(false)
}
override fun provideTestTemplateInvocationContexts(context: ExtensionContext): Stream<TestTemplateInvocationContext> {
val annotation = AnnotationSupport.findAnnotation(
context.testMethod.orElseThrow { ExtensionContextException("Must be annotated on method") },
Retry::class.java
).orElseThrow { ExtensionContextException("${Retry::class.java} not found on method") }
checkValidRetry(annotation)
return IntStream.rangeClosed(1, annotation.invocationCount)
.mapToObj { RetryTemplateContext(it, annotation.invocationCount, annotation.minSuccess) }
}
private fun checkValidRetry(annotation: Retry) {
if (annotation.invocationCount < 1) {
throw ExtensionContextException("${annotation.invocationCount} must be greater than or equal to 1")
}
if (annotation.minSuccess < 1 || annotation.minSuccess > annotation.invocationCount) {
throw ExtensionContextException("Invalid ${annotation.minSuccess}")
}
}
}
data class representing the retry (injected into test cases in this example using ParameterResolver).data class RetryInfo(val invocation: Int, val maxInvocations: Int)
Exception用于表示失败的重试:Exception used for representing failed retries:import java.lang.Exception
internal class RetryingTestFailure(invocation: Int, cause: Throwable) : Exception("Failed test execution at invocation #$invocation", cause)
ExecutionCondition，ParameterResolver和TestExecutionExceptionHandler的主扩展.import org.junit.jupiter.api.extension.ConditionEvaluationResult
import org.junit.jupiter.api.extension.ExecutionCondition
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.ParameterContext
import org.junit.jupiter.api.extension.ParameterResolver
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler
import org.opentest4j.TestAbortedException
internal class RetryingTestExecutionExtension(
private val invocation: Int,
private val maxInvocations: Int,
private val minSuccess: Int
) : ExecutionCondition, ParameterResolver, TestExecutionExceptionHandler {
override fun evaluateExecutionCondition(
context: ExtensionContext
): ConditionEvaluationResult {
val failureCount = getFailures(context).size
// Shift -1 because this happens before test
val successCount = (invocation - 1) - failureCount
when {
(maxInvocations - failureCount) < minSuccess -> // Case when we cannot hit our minimum success
return ConditionEvaluationResult.disabled("Cannot hit minimum success rate of $minSuccess/$maxInvocations - $failureCount failures already")
successCount < minSuccess -> // Case when we haven't hit success threshold yet
return ConditionEvaluationResult.enabled("Have not ran $minSuccess/$maxInvocations successful executions")
else -> return ConditionEvaluationResult.disabled("$minSuccess/$maxInvocations successful runs have already ran. Skipping run $invocation")
}
}
override fun supportsParameter(
parameterContext: ParameterContext,
extensionContext: ExtensionContext
): Boolean = parameterContext.parameter.type == RetryInfo::class.java
override fun resolveParameter(
parameterContext: ParameterContext,
extensionContext: ExtensionContext
): Any = RetryInfo(invocation, maxInvocations)
override fun handleTestExecutionException(
context: ExtensionContext,
throwable: Throwable
) {
val testFailure = RetryingTestFailure(invocation, throwable)
val failures: MutableList<RetryingTestFailure> = getFailures(context)
failures.add(testFailure)
val failureCount = failures.size
val successCount = invocation - failureCount
if ((maxInvocations - failureCount) < minSuccess) {
throw testFailure
} else if (successCount < minSuccess) {
// Case when we have still have retries left
throw TestAbortedException("Aborting test #$invocation/$maxInvocations- still have retries left",
testFailure)
}
}
private fun getFailures(context: ExtensionContext): MutableList<RetryingTestFailure> {
val namespace = ExtensionContext.Namespace.create(
RetryingTestExecutionExtension::class.java)
val store = context.parent.get().getStore(namespace)
@Suppress("UNCHECKED_CAST")
return store.getOrComputeIfAbsent(context.requiredTestMethod.name, { mutableListOf<RetryingTestFailure>() }, MutableList::class.java) as MutableList<RetryingTestFailure>
}
}
import org.junit.jupiter.api.DisplayName
internal class MyRetryableTest {
@DisplayName("Fail all retries")
@Retry(invocationCount = 5, minSuccess = 3)
internal fun failAllRetries(retryInfo: RetryInfo) {
println(retryInfo)
throw Exception("Failed at $retryInfo")
}
@DisplayName("Only fail once")
@Retry(invocationCount = 5, minSuccess = 4)
internal fun succeedOnRetry(retryInfo: RetryInfo) {
if (retryInfo.invocation == 1) {
throw Exception("Failed at ${retryInfo.invocation}")
}
}
@DisplayName("Only requires single success and is first execution")
@Retry(invocationCount = 5, minSuccess = 1)
internal fun firstSuccess(retryInfo: RetryInfo) {
println("Running: $retryInfo")
}
@DisplayName("Only requires single success and is last execution")
@Retry(invocationCount = 5, minSuccess = 1)
internal fun lastSuccess(retryInfo: RetryInfo) {
if (retryInfo.invocation < 5) {
throw Exception("Failed at ${retryInfo.invocation}")
}
}
@DisplayName("All required all succeed")
@Retry(invocationCount = 5, minSuccess = 5)
internal fun allRequiredAllSucceed(retryInfo: RetryInfo) {
println("Running: $retryInfo")
}
@DisplayName("Fail early and disable")
@Retry(invocationCount = 5, minSuccess = 4)
internal fun failEarly(retryInfo: RetryInfo) {
throw Exception("Failed at ${retryInfo.invocation}")
}
}
TestExecutionExceptionHandler.handleTestExecutionException抛出TestAbortedException是否应该中止测试，但是我在这里使用它.TestAbortedException from the TestExecutionExceptionHandler.handleTestExecutionException is supposed to abort the test, but I am using it here.
