如何基于Testng测试的通过率阈值使Jenkins工作失败 [英] How to fail a Jenkins job based on pass rate threshold of testng tests

问题描述

我们正在尝试将Jenkins用作我们的CI/CD解决方案，并且如果我们的TestNG测试未达到设置的通过率阈值，我们希望能够使部署失败.

We are trying to use Jenkins as our CI/CD solution, and want ability to fail a deployment if our TestNG tests do not meet a set pass rate threshold.

当前，我们的Jenkins作业通过maven执行我们的TestNG测试，因此任何测试失败都将构建标记为失败. IE.任何低于100％的通过率，都会使构建作业失败.

Currently our Jenkins job executes our TestNG tests via maven, thus any test failure marks the build as failed. I.e. anything less than 100% pass rate, will fail the build job.

我们希望能够对其进行配置-即，如果通过率低于(例如)98％，则使测试作业失败.

We want ability to make it configurable - i.e. fail the test job if pass rate is below, say, 98%.

这是我可以通过某些插件在Jenkins作业内部进行控制或配置的东西，还是我必须为TestNG侦听器内部进行编码的东西，例如onFinish()并根据计算出的通过率设置返回值?

Is this something i can control or configure inside the Jenkins job via some plugin, or is it something i have to code for inside TestNG listeners, i.e. onFinish() and set the return value based on the calculated pass rate?

推荐答案

我假设您的测试用例有些脆弱，即使有2％的失败，也会使您通过构建.否则，我会说所有低于100％的内容都是构建失败.

I'm assuming there are some flakiness in your test cases that is causing you to make the build pass even when there are 2% failures? Otherwise, I'd say anything less than 100% is a build failure.

话虽如此，如何使用声纳?

That being said, how about using Sonar?

使用它，当单元测试成功为<时，构建可能会失败. 98％(请参见此处的示例屏幕截图).

With it, you can fail the build when the unit test success is < 98% (see some example screenshots here).

您也可以在其他情况下使构建失败.例如，当承保范围下降到一定数量以下或发现严重问题时.另外，您将获得包含所有与构建相关的指标的仪表板的额外好处.

You can also fail the build on other occasions as well. For example, when the coverage drops below a certain amount or when a critical issue is found. Plus, you'll get the added bonus of a dashboard that contains all your build related metrics.

您将需要下载名为 Build Breaker 的插件，该插件将使构建失败当质量门低于某个阈值时.

You will need to download a plugin called Build Breaker that will fail the build when a quality gate is below a certain threshold.

要注意的一件事是Sonarqube的人们建议您请勿使用此插件.因此，您还可以执行查询Sonar rest api 之类的操作，使构建失败，或者等到它们

One thing to note is that the Sonarqube folks are recommending that you don't use this plugin. Thus, you can also do things like query the Sonar rest api and fail the build or wait till they add core support for it.

这篇关于如何基于Testng测试的通过率阈值使Jenkins工作失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！