如何快速增加计数器Cassandra w / o陈旧 [英] how to rapidly increment counters in Cassandra w/o staleness

问题描述

我有一个Cassandra问题。你知道Cassandra如何更新/增加计数器吗？

I have a Cassandra question. Do you know how Cassandra does updates/increments of counters?

我想使用暴风雨（CassandraCounterBatchingBolt from storm-contrib repo on github）写入cassandra。但是，我不知道incrementCounterColumn（）方法的一些实现是如何工作的..还有cassandra计数器的限制（从： http://wiki.apache.org/cassandra/Counters ），这使得它们对我的情景IMHO无用：

I want to use a storm bolt (CassandraCounterBatchingBolt from storm-contrib repo on github) which writes into cassandra. However, I'm not sure how some of the implementation of the incrementCounterColumn() method works .. and there is also the limitations with cassandra counters (from: http://wiki.apache.org/cassandra/Counters) which makes them useless for my scenario IMHO:

• 如果写操作意外失败（超时或失去与协调器节点的连接），客户端将不知道操作是否已执行。重试可能导致计数超过CASSANDRA-2495。

• If a write fails unexpectedly (timeout or loss of connection to the coordinator node) the client will not know if the operation has been performed. A retry can result in an over count CASSANDRA-2495.

计数器移除本质上受到限制。例如，如果您非常快速地执行序列increment，remove，increment，则可能会丢失删除

Counter removal is intrinsically limited. For instance, if you issue very quickly the sequence "increment, remove, increment" it is possible for the removal to be lost

无论如何，这里是我的场景：

我更新相同的计数器比更新传播到其他Cassandra节点。

Anyway, here is my scenario:
I update the same counter faster than the updates propagate to other Cassandra nodes.

示例：

说我有3个Cassandra节点。每个节点上的计数器为0.

Node1：0，node2：0，node3：0

Example:
Say I have 3 cassandra nodes. The counters on each of these nodes are 0.
Node1:0, node2:0, node3:0

Node1：0，node2：0，node3：0

An increment comes: 5 -> Node1:0, node2:0, node3:0

增量从节点2开始 - 仍然需要传播到node1和node3

Node1：0 ，node2：5，node3：0

Increment starts at node 2 – still needs to propagate to node1 and node3
Node1:0, node2:5, node3:0

同时，另一个增量在上一个增量之前到达

被传播：3 - > Node1： node2：5，node3：0

In the meantime, another increment arrives before previous increment
is propagated: 3 -> Node1:0, node2:5, node3:0

假设3在不同的节点开始，我们有：

Node1：3，node2：5 ，node3：0

Assuming 3 starts at a different node than where 5 started we have:
Node1:3, node2:5, node3:0

现在如果3被传播到其他节点AS AN INCREMENT而不是新值
和相同的5），然后最终节点都将等于8，这是我想要的。

Now if 3 gets propagated to the other nodes AS AN INCREMENT and not as a new value (and the same for 5) then eventually the nodes would all equal 8 and this is what I want.

如果3覆盖5（因为它有一个更新的时间戳）这是有问题的，而不是我想要的。

If 3 overwrites 5 (because it has a later timestamp) this is problematic – not what I want.

您知道Cassandra如何处理这些更新/增量吗？

请注意，在写入之前的读取仍然容易受到同样的影响问题取决于从哪个副本节点读取执行（如果传播不远，Quorum仍然可能失败）

Note, that a read before a write is still susceptible to the same problem depending from which replica node the read executes (Quorum can still fail if propagation is not far along)

我也认为可能放一个缓存b / w我的

I'm also thinking that maybe putting a cache b/w my storm bolt and Cassandra might solve this issue but that's a story for another time.

推荐答案

C *中的计数器有一个复杂的内部表示避免了在无领导分布式系统中计数事物的大多数（但不是全部）问题。我喜欢把他们当成一个破碎的柜台。计数器由通过主机ID和版本号标识的多个子计数器组成。接收计数器操作的主机仅增加其自身的子计数器，并且还增加版本。然后它将其整个计数器状态复制到其他副本，这将其与它们的状态合并。当读取计数器时，处理读操作的节点通过总计每个主机的计数总数来确定计数器值。

Counters in C* have a complex internal representation that avoids most (but not all) problems of counting things in a leaderless distributed system. I like to think of them as sharded counters. A counter consists of a number of sub-counters identified by host ID and a version number. The host that receives the counter operation increments only its own sub-counter, and also increments the version. It then replicates its whole counter state to the other replicas, which merge it with their states. When the counter is read the node handling the read operation determines the counter value by summing up the total of the counts from each host.

在每个节点上，计数器增量只是像Cassandra中的一切，只是一个写作。增量写入到memtable中，并且通过合并来自memtable和所有SSTables的所有增量，在读取时确定本地值。

On each node a counter increment is just like everything else in Cassandra, just a write. The increment is written to the memtable, and the local value is determined at read time by merging all of the increments from the memtable and all SSTables.

我希望解释有助于你相信我，当我说，你不必担心增加计数器比Cassandra可以处理。由于每个节点保持其自己的计数器，并且从不复制增量操作，因此不会由于诸如读 - 修改 - 写方案的竞争状况而引起计数丢失的可能性。如果Cassandra接受写入，你几乎可以保证它会计数。

I hope that explanation helps you believe me when I say that you don't have to worry about incrementing counters faster than Cassandra can handle. Since each node keeps its own counter, and never replicates increment operations, there is no possibility of counts getting lost by race conditions like a read-modify-write scenario would introduce. If Cassandra accepts the write, your're pretty much guaranteed that it will count.

你不能保证，但是，计数将正确显示次。如果一个增量写入一个节点，但计数器值从另一个节点读取，则不能保证增量已被复制，您还必须考虑在网络分区期间将发生什么。这或多或少与Cassandra中的任何写入相同，它的最终一致的性质，这取决于您用于操作的一致性级别。

What you're not guaranteed, though, is that the count will appear correct at all times unless. If an increment is written to one node but the counter value read from another just after, there is not guarantee that the increment has been replicated, and you also have to consider what would happen during a network partition. This more or less the same with any write in Cassandra, it's in its eventually consistent nature, and it depends on which consistency levels you used for the operations.

丢失确认的可能性。如果你做一个增量并松开与Cassandra的连接，然后你可以得到回复，你不能知道你的写作是否有。当你得到连接回来，你不能告诉任何一个，因为你不知道什么计数是你增加之前。这是在一致性上选择可用性的系统的固有问题，以及您为许多其他好处支付的价格。

There is also the possibility of a lost acknowledgement. If you do an increment and loose the connection to Cassandra before you can get the response back you can't know whether or not your write got though. And when you get the connection back you can't tell either, since you don't know what the count was before you incremented. This is an inherent problem with systems that choose availability over consistency, and the price you pay for many of the other benefits.

最后，快速删除，增量，删除是真实的，你应该避免的东西。问题是增量操作将基本上复活列，并且如果这些操作彼此足够接近，它们可能获得相同的时间戳。 Cassandra是严格的最后写操作，根据操作的时间戳最后确定。如果两个操作具有相同的时间戳，则更大的一个胜利，这意味着以严格的字节顺序排序的一个。这是真的，但我不会担心太多，除非你做非常快速的写入和删除到相同的值（这可能是你的数据模型中的一个错误）。

Finally, the issue of rapid remove, increment, removes are real, and something you should avoid. The problem is that the increment operation will essentially resurrect the column, and if these operations come close enough to each other they might get the same timestamp. Cassandra is strictly last-write-wins and determines last based on the timestamp of the operation. If two operations have the same time stamp, the "greater" one wins, which means the one which sorts after in a strict byte order. It's real, but I wouldn't worry too much about it unless you're doing very rapid writes and deletes to the same value (which is probably a fault in your data model).

这里是Cassandra计数器内部的一个很好的指南： http：// www .datastax.com / wp-content / uploads / 2011/07 / cassandra_sf_counters.pdf

Here's a good guide to the internals of Cassandra's counters: http://www.datastax.com/wp-content/uploads/2011/07/cassandra_sf_counters.pdf

这篇关于如何快速增加计数器Cassandra w / o陈旧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！