持续集成中的性能测试? [英] Performance testing in continuous integration?

问题描述

我们已经有一个持续集成流程，用于构建、运行单元测试、进行静态代码分析和生成文档.但是，我们希望将其扩展为包括自动性能测试.在本例中，我们正在开发一个 .NET 网络应用程序.

We already have a continuous integration process going where we build, run unit tests, do static code analysis and generate documentation. However, we would like expand this to include automatic performance testing. In this case, we are working on a .NET web application.

我们已经使用 JMeter 进行了一些性能测试(在 CI 过程之外)，但我不知道这是否是包含在 CI 过程中的最佳工具?硒是一种选择吗?WAPT 专业版?

We have done some performance testing with JMeter (outside the CI process), but I don't know if this is the best tool to include in a CI process? Is Selenium an option? WAPT Pro?

我们应该在哪些级别上测试性能?我们应该有一套性能单元测试"吗?我们是否应该在类似生产的环境中运行 JMeter(或类似的东西)并在任何请求耗时 > 1 秒时失败?这样的东西不会有太大的方差吗?

On which levels should we test the performance? Should we have a set of "performance unit tests"? Should we run JMeter (or something similar) on a production-like environment and fail if any requests takes > 1 second? Wouldn't something like this have too high variance?

那么，你们是否将自动性能测试作为 CI 的一部分?你测试什么，你使用哪些工具?你的经历是怎样的?

So, do you guys include automatic performance testing as part of your CI? What do you test, and which tools do you use? What has your experience been like?

推荐答案

首先，JMeter 是包含在 CI 中的不错选择，因为它可以从命令行运行，并且您可以在执行此操作时传入变量.我会推荐它来完成这项任务.

First off, JMeter is a good choice for inclusion in CI because it can be run from the command line and you can pass in variables when you do this. I would recommend it for this task.

一般来说，集成 Perf.测试 CI 很困难.您已经列出了许多原因，所以您已经成功了一半，因为您了解这些限制.这就是问题所在:Perf 是可能的.在 CI 中进行测试，但仅限于有限的范围.

In general though, integrating Perf. testing into CI is difficult. You've already listed many of the reasons why this is so you're already half way there because you understand the limitations. And that's the rub: it IS possible to have Perf. tests in CI but only to a limited extent.

我认为一个好的方法遵循以下一些原则:

I think that a good approach follows some of these principles:

您无法在 CI 中运行满载(或浸泡或容量)测试，这是不切实际的.结果是主观的，需要人工解释，运行测试需要时间.但是您可以运行一组更简单、更精简的测试来测量请求的响应时间，然后您可以评估这些响应时间:

You can't run full load (or soak or capacity) tests in CI, it's not practical. The results are subjective and need human interpretation and it takes time to run the tests. But you can run a simpler, cut down set of tests that measure response times for requests and then you can evaluate these response times either:

• 针对 NFR 或预期范围 - 即.应该少于 1 秒.
• 与之前的结果相反 - 即.与上次构建的偏差不应超过 10%.

您还可以运行自动加载/性能.测试 - 全量 - 在构建过程之外.'半CI'.那么，也许您可​​以自动化测试运行一夜，然后在早上检查结果?

You can also run automated load / perf. tests - at full volume - outside of the build process. 'Semi CI'. So maybe you could automate a test to run overnight and then check the results in the morning?

迭代.只需开始这样做并获得结果并微调测试以及随着时间的推移如何解释它们.保持简单并专注于看起来有用的领域.不要大张旗鼓地启动，保持安静，直到你对这个过程有信心，然后开始构建失败并告诉人们它 - 最初，你可能会得到很多误报.

Iterate. Just start doing it and getting results and fine tune the tests and how you interpret them over time. Keep it simple and focus on the areas that appear to be useful. Don't launch with a fanfare, keep it quiet until you have confidence with the process and then start failing builds and telling people about it - initially, you're likely to get lots of false negatives.

检测结果做这个.很多.CI 就是尽早失败，因此如果您将其视为最终目标，那么实现它的最佳方法是尽早并经常运行测试，但这样做的问题是您可能会被数据淹没.因此，一种处理数据并呈现相关信息的有效方法有很大帮助.

Instrument your results Do this. A lot. CI is all about failing early so if you take that as you end objective then the best way to achieve it is to run tests early and often but the problem with that is you risk getting buried in data. So an effective method to crunch the data and present the relevant information helps considerably.

您无法将整个过程自动化到红旗绿旗 - 但您应该尽可能地沿着这条路走下去.

You can't automate the whole process down to Red Flag Green Flag - but you should try to get as far down that path as possible.

最后，领队 Perf 作了精彩的演讲.谷歌的测试人员涵盖了这个主题.现在有点老了，但原则仍然存在.另外，几周后我要去聚会，Channel4，英国媒体公司将讨论他们是如何处理这个问题的 - 也许你可以索要一些幻灯片.

Finally, there was a very good talk given by the lead Perf. tester at Google that covers this subject. It's a bit old now but the principles still stand. Plus, in a few weeks I'm going to a meetup where Channel4, a British media company, will be talking about how they approached this - maybe you can ask for some slides.

这篇关于持续集成中的性能测试?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！