cassandra，在添加新节点期间建立的二级索引永远持续 [英] cassandra, secondary indexes building during adding of a new node lasts forever

问题描述

我试图添加新节点到我们的群集（cassandra 2.1.11,16个节点，32Gb ram，2x3Tb hdd，8core cpu，1个数据中心，2个机架，每个节点上约700Gb的数据）。新节点启动后，来自16个现有节点的数据（大约600Gb）成功地转移到新节点，并开始建立辅助索引。二级索引构建的过程看起来很正常，我看到了关于一些二级索引构建和一些流任务的成功完成的信息：

  INFO [StreamReceiveTask：9] 2015-11-22 02：15：23,153 StreamResultFuture.java:180  -  [Stream＃856adc90-8ddd-11e5-a4be-69bddd44a709]与/192.168.21.66的会话完成
 
 INFO [StreamReceiveTask：9] 2015-11-22 02：15：23,152 SecondaryIndexManager.java:174  - 索引构建[docs.docs_ex_pl_ph_idx，docs.docs_lo_pl_ph_idx，docs.docs_author_login_idx，docs.docs_author_extid_idx，docs.docs_url_idx] complete 
  

Curently根据日志，16个流中有9个流成功完成。一切看起来不错，除了一个问题：这个过程已经持续5整天。

  nodetool compactionstats -H 
 
 code> 

显示

  build ... docs 882,4 MB 1,69 GB字节51,14％
  

一些建立索引的过程和它有一些进展，但非常缓慢，1个半小时左右。

新节点和任何现有节点是cassandra java进程有21k打开文件，而在任何现有节点上有300个打开文件，而在新节点上的数据目录中有80k个文件，而在任何现有节点上的数据目录中有300-500个文件。

是否正常？

解决方案

我知道这是一个老的问题，但我们碰到这个确切的问题与2.1.13使用DTCS。我们能够通过将memtable冲洗阈值增加到 0.7 来修复它在我们的测试环境中 - 这对我们没有任何意义，但可能值得尝试。 >

I'm trying to add new node to our cluster (cassandra 2.1.11, 16 nodes, 32Gb ram, 2x3Tb hdd, 8core cpu, 1 datacenter, 2 racks, about 700Gb of data on each node). After start of new node, data (approx 600Gb total) from 16 existing nodes successfully transfered to new node and building of secondary indexes starts. The process of secondary indexes building looks normal, i see info about successfull completition of some secondary indexes building and some stream tasks:

INFO  [StreamReceiveTask:9] 2015-11-22 02:15:23,153 StreamResultFuture.java:180 - [Stream #856adc90-8ddd-11e5-a4be-69bddd44a709] Session with /192.168.21.66 is complete

INFO  [StreamReceiveTask:9] 2015-11-22 02:15:23,152 SecondaryIndexManager.java:174 - Index build of [docs.docs_ex_pl_ph_idx, docs.docs_lo_pl_ph_idx, docs.docs_author_login_idx, docs.docs_author_extid_idx, docs.docs_url_idx] complete

Curently 9 out of 16 streams successfully finished, according to logs. Everything looks fine, except one issue: this process already lasts 5 full days. There is no errors in logs, no anything suspicious, except extremely slow progress.

nodetool compactionstats -H

shows

   Secondary index build   ...    docs    882,4 MB   1,69 GB   bytes     51,14%

So there is some process of index building and it has some progress, but very slow, 1% in half a hour or so.

The only significant difference between the new node and any of existing nodes is the fact that cassandra java process has 21k open files, in contrast of 300 open files on any existing node, and 80k files in the data dir on new node in contrast of 300-500 files in the data dir on any existing node.

Is it normal? At this speed it looks i'll spend 16 weeks or so to add 16 more nodes.

解决方案

I know this is an old question, but we ran into this exact issue with 2.1.13 using DTCS. We were able to fix it in our test environment by increasing memtable flush thresholds to 0.7 - which didn't make any sense to us, but may be worth trying.

这篇关于cassandra，在添加新节点期间建立的二级索引永远持续的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！