哪个测量正确，JMeter 还是 Apache ab? [英] Which gets the measurements right, JMeter or Apache ab?

问题描述

我开始在 JMeter 中编写一些基本测试，并惊讶于测量结果与 Apache ab 的测量结果如此不同.

I started writing some basic tests in JMeter and was surprised that the measurements are so different from those from Apache ab.

我有一个千兆 LAN，连接运行 Nginx 的 Intel i7 服务器和运行 JMeter 或 ab 的 i5 测试机.最初，我只是测试开箱即用的 Nginx 主页响应率.

I have a gigabit LAN connecting an Intel i7 server running Nginx and an i5 test machine running JMeter or ab. Initially, I am simply testing the out-of-the box Nginx home page response rate.

ab -c 1 -n 100 http://testserver.local/

给予

Document Path:          /
Document Length:        151 bytes

Concurrency Level:      1
Time taken for tests:   0.078 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      38400 bytes
HTML transferred:       15100 bytes
Requests per second:    1280.77 [#/sec] (mean)
Time per request:       0.781 [ms] (mean)
Time per request:       0.781 [ms] (mean, across all concurrent requests)
Transfer rate:          480.29 [Kbytes/sec] received

此结果始终可重现，+/- 几个百分点.

This result is consistently reproducible, +/- a few percent.

在 JMeter 中，我有一个 1-user 100-loop 线程组，其中包含:

In JMeter, I have a 1-user 100-loop thread group containing:

• 一个 HTTP 头管理器设置 Accept-Encoding: gzip
• 一个 HTTP 获取/采样器
• 总结报告监听器

只有 100 个样本，每次我运行它都会得到非常不一致的结果.但最令人吃惊的事实是，报告的吞吐量低至每秒 40 个请求(不是 1280 个).最高记录率是 1030，只有当我增加到 10,000 个样本时才达到这一点.

With only 100 samples, this gives wildly inconsistent results each time I run it. But the most startling fact is that the throughput is reported as low as 40 requests per second (not 1280). The highest recorded rate was 1030, and this was achieved only when I increased to 10,000 samples.

我认为 JMeter 是用于简单负载测试的错误工具是否正确，因为它的开销太高而无法进行准确测量?

Am I right in thinking that JMeter is the wrong tool for simple load tests because its overheads are too high to allow accurate measurements?

推荐答案

Jmeter 告诉您每个请求实际花费了多长时间.AB 只是做了一些非常基本的数学运算来获得整体平均值.因此，您的问题的直接答案是 jmeter 是正确的，而 ab 只是通过为您提供所有内容的平均值来粗略猜测.

Jmeter tells you how long each request actually took. AB just does some very basic math to get the overall average. So, the direct answer to your question is that jmeter gets it right and ab just makes a rough guess by giving you the mean across everything.

但是，当然，如果您将这两个工具并排放置并评估它们的速度，那么很明显 ab 的性能将超过 jmeter.Jmeter 只是做得更多，它记录更多数据并处理更多逻辑，因此处理单个请求需要更长的时间.一个简单的事实是，Jmeter 是一个功能齐全的负载测试工具，而 AB 则不是.

But, sure, if you put the two tools side by side and rate them for speed then it is clearly the case that ab is going to out perform jmeter. Jmeter just does more, it records more data and is processing more logic so it takes longer to turn around a single request. The simple fact is that Jmeter is a fully featured load testing tool, AB is, well, not.

问题是，负载测试工具的目标不是成为块上最快的孩子，而是关于能够构建应用程序负载类型的真实表现当它上线时可能会被击中.在这方面 jmeter 胜出，所以这真的取决于你的要求是什么.如果您只想使用最少的硬件生成尽可能多的请求，那么 ab 是一个不错的选择，但是如果您想构建一个具有代表性的测试，包括事务旅程、条件逻辑和各种其他有用的东西，那么 jmeter 是要走的路.可以这样想:它们都是 Apache 项目，但我认为 AB 旨在测试 apache Web 服务器，而 JMeter 旨在测试 Tomcat.

The thing is, the aim of a load testing tool is not to be the fastest kid on the block, instead it is about being able to build a realistic representation of the sort of load your app might be hit with when it goes live. In this respect jmeter wins hands down, so it really depends on what your requirements are. If you just want to generate as many requests as possible using the least amount of hardware then ab is a nice choice but if you want to build a representative test, with transactional journeys, conditional logic and all sorts of other useful stuff, then jmeter is the way to go. Think of it like this: they are both Apache projects but AB was, I think, designed to test the apache web server, JMeter, however, was designed to test Tomcat.

现在，我猜测 jmeter 产生了不一致的结果，因为它在运行的机器上达到了限制.我敢打赌你在 GUI 模式下运行并且至少有一个侦听器处于活动状态，这样你就要求该工具做很多事情.如果您需要高请求率，那么 Jmeter 具有精益和平均模式.通常，对于大容量，最佳实践是在命令行中以很少的侦听器执行测试；apache jmeter 站点上有很多关于这个主题的信息.

Now, I'm guessing that jmeter was producing inconsistent results because it was hitting a limit on the machine it was running on. I'm betting you were running in GUI mode and had at least one listener active, like this you are asking the tool to do a lot. If you need a high rate of requests then Jmeter has a lean and mean mode. Typically, for large volumes the best practice is to execute tests at the command line with very few listeners; there's lots of info about this subject on the apache jmeter site.

如果您真的要进行负载测试，您应该考虑的另一点是，为了真正从此类事情中受益，您需要首先确定您的站点需要支持哪种负载，然后你应该设计一个代表这个的测试.这是通过使用起搏和模拟等待时间来实现的.告诉一个线程应该离开并尽可能快地运行的问题是它会在其本地条件允许的情况下尽可能快地迭代，但是总会有一些东西总是休息时间，即使是 ab 也是有限的；无论工具多么轻量级，它仍然可以做一些事情.但是，如果您调整请求的速度，那么您就可以消除这个问题，并且作为一个相当有用的额外奖励，您最终会在运行之间和代码构建之间保持一致性，因此即使您的服务器加速或减速(更改代码库)您的测试仍会发出相同的请求率 - 这对于基准测试非常有用.

Another point you should consider, if you're really getting into load testing, is that in order to really get benefit from this sort of thing you need to first decide what sort of load you need your site to support and only then should you design a test that represents this. This is achieved using pacing and simulated wait times. The problem with telling a thread that is should just go away and run as fast as it possibly can is that it will iterate as fast as its local conditions allow it to, but there will always be something that puts the breaks on, even ab is limited; no matter how lightweight a tool is it still does something. But if you pace your requests then you remove this problem and as a rather useful added bonus you end up with consistency between runs and between builds of the code, so even if your server speeds up or slows down (with changes to the code base) your test will still make the same rate of requests - which is pretty useful for benchmarking.

如果您想进一步了解 JMeter，请查看恒定吞吐量计时器，然后使用多个线程来构建您需要表示的流量级别.

If you want to take JMeter further then have a look at the Constant Throughput Timer and then use multiple threads to build the level of traffic you need to represent.

这篇关于哪个测量正确，JMeter 还是 Apache ab?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！