完全删除已停用的Cassandra节点 [英] Fully removing a decommissioned Cassandra node

问题描述

运行Cassandra 1.0，我将一个戒指从5个节点缩减到4.为了做到这一点，我在要删除的节点上运行 nodetool decommission 停止该主机上的cassandra，并使用 nodetool move 和 nodetool cleanup 更新剩余4个节点上的标记，重新平衡集群。

Running Cassandra 1.0, I am shrinking a ring from 5 nodes down to 4. In order to do that I ran nodetool decommission on the node I want to remove, then stopped cassandra on that host and used nodetool move and nodetool cleanup to update the tokens on the remaining 4 nodes to rebalance the cluster.

我的种子节点是A和B.我删除的节点是C.

My seed nodes are A and B. The node I removed is C.

工作良好6-7天，但现在我的四个节点之一认为退役的节点仍然是环的一部分。

That seemed to work fine for 6-7 days, but now one of my four nodes thinks the decommissioned node is still part of the ring.

为什么这样发生，什么是正确的

Why did this happen, and what's the proper way to fully remove the decommissioned node from the ring?

这是节点上的 nodetool ring 的输出仍然认为退役节点是环的一部分：

Here's the output of nodetool ring on the one node that still thinks the decommissioned node is part of the ring:

Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               127605887595351923798765477786913079296     
xx.x.xxx.xx     datacenter1 rack1       Up     Normal  616.17 MB       25.00%  0                                           
xx.xxx.xxx.xxx  datacenter1 rack1       Up     Normal  1.17 GB         25.00%  42535295865117307932921825928971026432      
xx.xxx.xx.xxx   datacenter1 rack1       Down   Normal  ?               9.08%   57981914123659253974350789668785134662      
xx.xx.xx.xxx    datacenter1 rack1       Up     Normal  531.99 MB       15.92%  85070591730234615865843651857942052864      
xx.xxx.xxx.xx   datacenter1 rack1       Up     Normal  659.92 MB       25.00%  127605887595351923798765477786913079296     

这是其他3个节点上 nodetool ring 的输出：

Here's the output of nodetool ring on the other 3 nodes:

Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               127605887595351923798765477786913079296     
xx.x.xxx.xx     datacenter1 rack1       Up     Normal  616.17 MB       25.00%  0                                           
xx.xxx.xxx.xxx  datacenter1 rack1       Up     Normal  1.17 GB         25.00%  42535295865117307932921825928971026432      
xx.xx.xx.xxx    datacenter1 rack1       Up     Normal  531.99 MB       25.00%  85070591730234615865843651857942052864      
xx.xxx.xxx.xx   datacenter1 rack1       Up     Normal  659.92 MB       25.00%  127605887595351923798765477786913079296     

更新： strong>
我尝试使用节点B上的 nodetool removetoken 删除节点，这是仍然声明节点C在环中的节点。该命令运行了5个小时，似乎没有做任何事情。唯一的变化是，当我在节点B上运行 nodetool ring 时，节点C的状态现在为离开。

UPDATE: I tried to remove the node using nodetool removetoken on node B, which is the one that still claims node C is in the ring. That command ran for 5 hours and didn't seem to do anything. The only change is that the state of Node C is "Leaving" now when I run nodetool ring on Node B.

推荐答案

我可以使用 nodetool removetoken 删除已停用的节点，但是我不得不使用 force

I was able to remove the decommissioned node using nodetool removetoken, but I had to use the force option.

这是命令的输出：

iowalker:~$ nodetool -h `hostname` removetoken 57981914123659253974350789668785134662

<waited 5 hours, the node was still there>

iowalker:~$ nodetool -h `hostname` removetoken status
RemovalStatus: Removing token (57981914123659253974350789668785134662). Waiting for replication confirmation from [/xx.xxx.xxx.xx,/xx.x.xxx.xx,/xx.xx.xx.xxx].
iowalker:~$ nodetool -h `hostname` removetoken force
RemovalStatus: Removing token (57981914123659253974350789668785134662). Waiting for replication confirmation from [/xx.xxx.xxx.xx,/xx.x.xxx.xx,/xx.xx.xx.xxx].
iowalker:~$ nodetool -h `hostname` removetoken status
RemovalStatus: No token removals in process.

这篇关于完全删除已停用的Cassandra节点的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！