Cassandra中的严重降级随着时间的推移连续流数据的写性能 [英] Severe degradation in Cassandra Write performance with continuous streaming data over time

问题描述

我注意到Cassandra写性能随时间的连续写入的严重退化。

我在一个宽列中插入时间戳（T）作为列名的时间序列数据，在一行中存储24小时的数据。
流数据是从数据生成器（4个实例，每个具有256个线程）中并行地将数据插入多行中写入。
此外，数据还插入到具有DateType和UUIDType索引的列系列中。

CF1：

  Col1 | Col2 | Col3（DateType）| Col（UUIDType4）| 
 

RowKey1

RowKey2

：

：

CF2

RowKey1（T1，V1）（T2，V3）（T4，V4）......

：

的插入数据点/秒随时间减少，直到不再有插入为止。初始性能为大约60000 ops / sec，大约6-8小时，然后逐渐减小到0 ops / sec。在所有节点上重新启动DataStax_Cassandra_Community_Server有助于恢复原始吞吐量，但在几个小时后再次观察行为。

操作系统：Windows Server 2008
节点数：5
Cassandra版本：DataStax社区1.2.3
RAM：8GB
HeapSize：3GB
垃圾收集器：默认设置[ParNewGC]

我也注意到no的增加。当性能开始下降时，OpsCenter报告的等待写入请求数（大约200,000）。

我无法理解什么是阻止写操作完成为什么他们随时间积累？我在Cassandra日志中没有看到任何可疑的东西。

操作系统设置与此有何关系？
进一步探讨此问题的任何建议

解决方案

使写入超时与2.0中的新默认值一致的2秒而不是10秒）将有助于您的写作积压，允许负载卸载更快地启动： https://issues.apache.org/jira/browse/CASSANDRA-6059

I notice a severe degradation in Cassandra write performance with continuous writes over time.

I am inserting time series data with time stamp (T) as the column name in a wide column that stores 24 hours worth of data in a single row. Streaming data is written from data generator (4 instances, each with 256 threads) inserting data into multiple rows in parallel. Additionally, data is also inserted into a column family that has indexes over DateType and UUIDType.

CF1:

        Col1    |   Col2    |   Col3(DateType)  |   Col(UUIDType4)  |   

RowKey1

RowKey2

:

:

CF2 (Wide column family):

RowKey1 (T1, V1) (T2, V3) (T4, V4) ......

RowKey2 (T1, V1) (T3, V3) .....

:

:

The no. of data points inserted/sec decreases over time until no further inserts are possible. The initial performance is of the order of 60000 ops/sec for ~6-8 hours and then it gradually tapers down to 0 ops/sec. Restarting the DataStax_Cassandra_Community_Server on all nodes helps restore the original throughput, but the behaviour is observed again after a few hours.

OS: Windows Server 2008 No.of nodes: 5 Cassandra version: DataStax Community 1.2.3 RAM: 8GB HeapSize: 3GB Garbage collector: default settings [ParNewGC]

I also notice a phenomenal increase in the no. of Pending write requests as reported by the OpsCenter (~of magnitude 200,000) when the performance begins to degrade.

I fail to understand what is preventing the write operations to be completed and why do they pile up over time? I do not see anything suspicious in the Cassandra logs.

Has the OS settings got anything to do with this? Any suggestions to probe this issue further?

解决方案

Bringing your write timeout in line with the new default in 2.0 (of 2s instead of 10s) will help with your write backlog by allowing load shedding to kick in faster: https://issues.apache.org/jira/browse/CASSANDRA-6059

这篇关于Cassandra中的严重降级随着时间的推移连续流数据的写性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！