Cassandra v。3，写入性能问题 [英] Cassandra v. 3, write performance issue

问题描述

我在使用Cassandra 3时遇到性能问题。

I have write performance issues with Cassandra 3.

我正在使用官方Docker镜像从以下位置试用Cassandra 3.3： https://github.com/docker-library/cassandra

I'm trying out Cassandra 3.3 using the official Docker image from here: https://github.com/docker-library/cassandra

I如下所示启动它：

docker run --net=host --rm cassandra:3.3

然后对它运行cassandra-stress：

Then run cassandra-stress against it:

cassandra-stress write

这为执行流量的四个线程提供了以下结果：

This gives me the following results for four threads executing traffic:

op rate                   : 1913 [WRITE:1913]
partition rate            : 1913 [WRITE:1913]
row rate                  : 1913 [WRITE:1913]
latency mean              : 2.1 [WRITE:2.1]
latency median            : 1.6 [WRITE:1.6]
latency 95th percentile   : 4.1 [WRITE:4.1]
latency 99th percentile   : 8.4 [WRITE:8.4]
latency 99.9th percentile : 20.5 [WRITE:20.5]
latency max               : 155.4 [WRITE:155.4]
Total partitions          : 154607 [WRITE:154607]
Total errors              : 0 [WRITE:0]
total gc count            : 13
total gc mb               : 1951
total gc time (s)         : 1
avg gc time(ms)           : 59
stdev gc time(ms)         : 28
Total operation time      : 00:01:20

使用Cassandra 2.2执行完全相同的操作官方图片：

Doing the exact same thing for Cassandra 2.2 using the official image:

docker run --net=host --rm cassandra:2.2

为我提供了四个线程的以下结果：

Gives me the following result with four threads:

op rate                   : 2248 [WRITE:2248]
partition rate            : 2248 [WRITE:2248]
row rate                  : 2248 [WRITE:2248]
latency mean              : 1.8 [WRITE:1.8]
latency median            : 1.4 [WRITE:1.4]
latency 95th percentile   : 3.5 [WRITE:3.5]
latency 99th percentile   : 7.2 [WRITE:7.2]
latency 99.9th percentile : 16.4 [WRITE:16.4]
latency max               : 129.5 [WRITE:129.5]
Total partitions          : 195461 [WRITE:195461]
Total errors              : 0 [WRITE:0]
total gc count            : 11
total gc mb               : 1612
total gc time (s)         : 1
avg gc time(ms)           : 62
stdev gc time(ms)         : 21
Total operation time      : 00:01:26

每秒写入数（3.3）比2.2少了15％。可能是什么原因呢？我一直在尝试更改各种参数，包括在上面在Oracle上运行Cassandra（与OpenJDK相比），没有太大的区别。我还尝试了不同版本的Cassandra 3，没有发现任何实际差异。我知道这是一个单节点，并且此基本测试的结果无法传输到生产设置。不过，我很想知道是否有人对此有解释或可以重做。

The number of writes/s is almost 15 % lower on 3.3 compared to 2.2. What could the reason for that be? I've been trying to change various parameters including running Cassandra on Oracle (compared to OpenJDK) above with no significant differences. I've also tried different versions of Cassandra 3 with no real difference spotted. I know that this is a single node and that the results from this basic test cannot be transferred to a production setting. Still, I'm curious to know if anybody has an explanation or can reproduce the behaviour.

欢迎任何输入！

更新2016-04-13 ，JVM参数差异：

diff jvm-param22-sorted jvm-param33-sorted

> -XX:+AlwaysPreTouch
# Before removing this from the 3.3 config:
# USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# cassand+     1 19.0 30.3 2876304 1229136 ?     Ssl  04:18   0:17 java
#
# After removing it:
# USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# cassand+   177 17.0  9.4 2885596 383972 ?      Sl+  04:59   0:17 java -
#
# The above could actually explain some of the issues that I've observed in 
# low memory environments with multiple containers running since I guess it
# means Cassandra will grab the whole heap at startup from the OS
# regardless of the actual need.    

# This was present twice in the 22 config. Should not matter.
< -XX:CMSWaitDuration=10000

# Removing this from the 3.3 config did not have any significant impact
> -XX:+ResizeTLAB

# Removing from the 3.3 config did not have any significant impact
> -XX:-UseBiasedLocking

# Adding this one to the 3.3 config did not have a significant impact
< -XX:+UseCondCardMark

推荐答案

我会自己回答备查。在我的案例中，罪魁祸首是JVM参数 -XX：+ AlwaysPreTouch ，它是Cassandra 3.0版本中默认添加的。删除它会使性能恢复到与2.2版相同。

I'll just answer myself for future reference. The culprit in my case was the JVM parameter -XX:+AlwaysPreTouch which was added as default in the 3.0 release of Cassandra. Removing it puts the performance back on par with the 2.2 release.

我主要在RAM数量相当有限的环境中进行了实验。我尚未对功能更强大的硬件进行任何实验，以查看该标志的作用。

I've mainly experimented in environments with fairly limited amount of RAM. I have not yet done any experiments on more powerful HW to see what the effects of this flag has there.

这篇关于Cassandra v。3，写入性能问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！