Cassandra GC需要30秒钟并挂起节点 [英] Cassandra GC takes 30 seconds and hangs node

查看：510
本文介绍了Cassandra GC需要30秒钟并挂起节点的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

环境



我有以下环境：




  • Cassandra 2.1.0

  • 一个DC中有5个节点，第二个DC中有4个节点

  • 每秒写2500次

  • 最小读数（通常不会，有时很少）





长时间运行的节点之后，GC开始花费更长时间，直到nodetool将此节点报告为down。



I显示gc.log并有以下屏幕截图：



JVM设置



注释后，Cassandra的完整命令行是：

  java -ea -javaagent：/ usr / lib / cassandra / bin /../ lib / jamm-0.2.6.jar 
  - XX：+ UseThreadPriorities 
 -XX：ThreadPriorityPolicy = 42 
 -Xms8192M 
 -Xmx8192M 
 -Xmn2048M 
 -XX：+ HeapDumpOnOutOfMemoryError 
 -Xss192k 
 -XX：+ UseParNewGC 
 -XX：+ UseConcMarkSweepGC 
 -XX：+ CMSParallelRemarkEnabled 
 -XX：SurvivorRatio = 8 
 -XX：MaxTenuringThreshold = 1 
 -XX ：CMSInitiatingOccupancyFraction = 75 
 -XX：+ UseCMSInitiatingOccupancyOnly 
 -XX：+ PrintGCDetails 
 -XX：+ PrintGCTimeStamps 
 -XX：+ PrintClassHistogram 
 -XX：+ PrintTenuringDistribution 
 -XX：+ PrintGCApplicationStoppedTime 
 -Xloggc：/var/log/cassandra/gc.log 
 -Djava.net.preferIPv4Stack = true 
 -Dcom.sun.management.jmxremote。 port = 8080 
 -Dcom.sun.management.jmxremote.ssl = false 
 -Dcom.sun.management.jmxremote.authenticate = false 
 -Dlogback.configurationFile = logback.xml 
 -Dcassandra.logdir = / usr / lib / cassandra / bin /../ logs 
 -Dcassandra.storagedir = / usr / lib / cassandra / bin /../ data 
 -Dcassandra-pidfile = /var/run/cassandra.pid 
 -cp /usr/lib/cassandra/bin/../conf:/usr/lib/cassandra/bin/../build/classes/main:/usr/lib /cassandra/bin/../build/classes/thrift:/usr/lib/cassandra/bin/../lib/airline-0.6.jar:/usr/lib/cassandra/bin/../lib/antlr- 3.2.jar：/ usr / lib / cassandra / bin /../ lib / apache-cassandra-2.1.0.jar：/ usr / lib / cassandra / bin /../ lib / apache-cassandra-clientutil-2.1。 0.jar：/ usr / lib / cassandra / bin /../ lib / apache-cassandra-thrift-2.1.0.jar：/ usr / lib / cassandra / bin /../lib/commons-cli- jar：/ usr / lib / cassandra / bin /../ lib / commons-codec-1.2.jar：/ usr / lib / cassandra / bin /../ lib / commons-lang3-3.1.jar：/ usr / lib /cassandra/bin/../lib/commons-math3-3.2.jar:/usr/lib/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/usr/lib/cassandra/bin /../lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/lib/cassandra/bin/../lib/disruptor-3.0.1.jar:/usr/lib/cassandra/bin/../lib/ guava-16.0.jar：/ usr / lib / cassandra / bin /../ lib / high-scale-lib-1.0.6.jar：/ usr / lib / cassandra / bin /../ lib / jackson- asl-1.9.2.jar：/ usr / lib / cassandra / bin /../ lib / jackson-mapper-asl-1.9.2.jar：/ usr / lib / cassandra / bin /../ lib / jamm- 0.2.6.jar：/ usr / lib / cassandra / bin /../ lib / javax.inject.jar：/ usr / lib / cassandra / bin /../ lib / jbcrypt-0.3m.jar：/ usr / lib / cassandra / bin /../ lib / jline-1.0.jar：/ usr / lib / cassandra / bin /../ lib / jna-4.0.0.jar：/ usr / lib / cassandra / bin / /lib/jna.jar:/usr/lib/cassandra/bin/../lib/json-simple-1.1.jar:/usr/lib/cassandra/bin/../lib/libthrift-0.9.1.jar ：/ usr / lib / cassandra / bin /../ lib / logback-classic-1.1.2.jar：/ usr / lib / cassandra / bin /../ lib / logback-core-1.1.2.jar： usr / lib / cassandra / bin /../ lib / lz4-1.2.0.jar：/ usr / lib / cassandra / bin /../ lib / metrics-core-2.2.0.jar：/ usr / lib / cassandra / bin /../ lib / netty-all-4.0.20.Final.jar：/ usr / lib / cassandra / bin /../ lib / reporter-config-2.1.0.jar：/ usr / lib / cassandra / bin /../ lib / slf4j-api-1.7.2.jar：/ usr / lib / cassandra / bin /../ lib / snakeyaml-1.11.jar：/ usr / lib / cassandra / bin / /lib/snappy-java-1.0.5.2.jar:/usr/lib/cassandra/bin/../lib/stream-2.5.2.jar:/usr/lib/cassandra/bin/../lib/super -csv-2.1.0.jar：/ usr / lib / cassandra / bin /../ lib / thrift-server-0.3.5.jar org.apache.cassandra.service.CassandraDaemon 
  
 
 
 
 问题 
 
 
 
问题是什么？此外，我最近有
 
 
 com.google.common.util.concurrent.UncheckedExecutionException：
 java.lang.RuntimeException：
 org.apache.cassandra.exceptions.ReadTimeoutException：Operation timed 
 out  -  received only 2 responses 
 
 
 
 
解决方案
如果你有一个繁重的工作量，你会得到很多长的小GC，有一个2GB的新基因。您可能希望将其拨回800mb。
 
Environment



I have a following environment:



    

  • Cassandra 2.1.0
    • 

  • 5 nodes in one DC, 4 nodes in second DC
    • 

  • 2500 writes per seconds
    • 

  • minimal reads (usually none, sometimes few)
    • 




Problem



After a long running node, GC starts to take longer and longer, until the nodetool reports this node as down.



I visualize gc.log and have a following screenshot:




JVM settings



After comment the full command line of working Cassandra is:
java -ea -javaagent:/usr/lib/cassandra/bin/../lib/jamm-0.2.6.jar
 -XX:+UseThreadPriorities
 -XX:ThreadPriorityPolicy=42
 -Xms8192M
 -Xmx8192M
 -Xmn2048M
 -XX:+HeapDumpOnOutOfMemoryError
 -Xss192k
 -XX:+UseParNewGC
 -XX:+UseConcMarkSweepGC
 -XX:+CMSParallelRemarkEnabled
 -XX:SurvivorRatio=8
 -XX:MaxTenuringThreshold=1
 -XX:CMSInitiatingOccupancyFraction=75
 -XX:+UseCMSInitiatingOccupancyOnly
 -XX:+PrintGCDetails
 -XX:+PrintGCTimeStamps
 -XX:+PrintClassHistogram
 -XX:+PrintTenuringDistribution
 -XX:+PrintGCApplicationStoppedTime
 -Xloggc:/var/log/cassandra/gc.log
 -Djava.net.preferIPv4Stack=true
 -Dcom.sun.management.jmxremote.port=8080
 -Dcom.sun.management.jmxremote.ssl=false
 -Dcom.sun.management.jmxremote.authenticate=false
 -Dlogback.configurationFile=logback.xml
 -Dcassandra.logdir=/usr/lib/cassandra/bin/../logs
 -Dcassandra.storagedir=/usr/lib/cassandra/bin/../data
 -Dcassandra-pidfile=/var/run/cassandra.pid
 -cp /usr/lib/cassandra/bin/../conf:/usr/lib/cassandra/bin/../build/classes/main:/usr/lib/cassandra/bin/../build/classes/thrift:/usr/lib/cassandra/bin/../lib/airline-0.6.jar:/usr/lib/cassandra/bin/../lib/antlr-3.2.jar:/usr/lib/cassandra/bin/../lib/apache-cassandra-2.1.0.jar:/usr/lib/cassandra/bin/../lib/apache-cassandra-clientutil-2.1.0.jar:/usr/lib/cassandra/bin/../lib/apache-cassandra-thrift-2.1.0.jar:/usr/lib/cassandra/bin/../lib/commons-cli-1.1.jar:/usr/lib/cassandra/bin/../lib/commons-codec-1.2.jar:/usr/lib/cassandra/bin/../lib/commons-lang3-3.1.jar:/usr/lib/cassandra/bin/../lib/commons-math3-3.2.jar:/usr/lib/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/usr/lib/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/lib/cassandra/bin/../lib/disruptor-3.0.1.jar:/usr/lib/cassandra/bin/../lib/guava-16.0.jar:/usr/lib/cassandra/bin/../lib/high-scale-lib-1.0.6.jar:/usr/lib/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/lib/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/lib/cassandra/bin/../lib/jamm-0.2.6.jar:/usr/lib/cassandra/bin/../lib/javax.inject.jar:/usr/lib/cassandra/bin/../lib/jbcrypt-0.3m.jar:/usr/lib/cassandra/bin/../lib/jline-1.0.jar:/usr/lib/cassandra/bin/../lib/jna-4.0.0.jar:/usr/lib/cassandra/bin/../lib/jna.jar:/usr/lib/cassandra/bin/../lib/json-simple-1.1.jar:/usr/lib/cassandra/bin/../lib/libthrift-0.9.1.jar:/usr/lib/cassandra/bin/../lib/logback-classic-1.1.2.jar:/usr/lib/cassandra/bin/../lib/logback-core-1.1.2.jar:/usr/lib/cassandra/bin/../lib/lz4-1.2.0.jar:/usr/lib/cassandra/bin/../lib/metrics-core-2.2.0.jar:/usr/lib/cassandra/bin/../lib/netty-all-4.0.20.Final.jar:/usr/lib/cassandra/bin/../lib/reporter-config-2.1.0.jar:/usr/lib/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/usr/lib/cassandra/bin/../lib/snakeyaml-1.11.jar:/usr/lib/cassandra/bin/../lib/snappy-java-1.0.5.2.jar:/usr/lib/cassandra/bin/../lib/stream-2.5.2.jar:/usr/lib/cassandra/bin/../lib/super-csv-2.1.0.jar:/usr/lib/cassandra/bin/../lib/thrift-server-0.3.5.jar org.apache.cassandra.service.CassandraDaemon 

Question



What should I check to find out what the problem is? Moreover I have recent 

  
com.google.common.util.concurrent.UncheckedExecutionException:
  java.lang.RuntimeException:
  org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed
  out - received only 2 responses




 解决方案 
If you've got a write heavy workload, you're going to end up with lots of long minor GCs with a 2gb new gen.  You'll probably want to dial that back to 800mb.


                        
这篇关于Cassandra GC需要30秒钟并挂起节点的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
相关文章
Java开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆