达到墓碑限制时会发生什么 [英] What exactly happens when tombstone limit is reached

问题描述

根据cassandra的日志(请参阅下文)，由于存在过多的tombstones，查询将被中止.发生这种情况是因为我每周都会使用太低的计数器来清理(删除)行.

According to cassandra's log (see below) queries are getting aborted due to too many tombstones being present. This is happening because once a week I cleanup (delete) rows with a counter that is too low. This 'deletes' hundreds of thousands of rows (marks them as such with a tombstone.)

如果在此表中重新出现已删除的行是完全没有问题的，因为在清理过程中某个节点处于关闭状态，所以我将单个受影响的表的gc grace time设置为10小时(从默认为10天)，因此可以快速地将已删除逻辑删除的行永久删除.

It is not at all a problem if, in this table, a deleted row re-appears because a node was down during the cleanup process, so I set the gc grace time for the single affected table to 10 hours (down from default 10 days) so the tombstoned rows can get permanently deleted relatively fast.

无论如何，我必须将tombstone_failure_threshold设置得很高，以避免出现以下异常. (从十万个增加到一亿个.)我的问题是，这有必要吗?我完全不知道哪种类型的查询会中止.插入，选择，删除?

Regardless, I had to set the tombstone_failure_threshold extremely high to avoid the below exception. (one hundred million, up from one hundred thousand.) My question is, is this necessary? I have absolutely no idea what type of queries get aborted; inserts, selects, deletes?

如果只是某些选择被中止，那没什么大不了的.但这是假设中止意味着上限"，因为查询会过早停止并返回在找到太多墓碑之前设法收集的所有实时数据.

If it's merely some selects being aborted, it's not that big a deal. But that's assuming abort means 'capped' in that the query stops prematurely and returns whatever live data it managed to gather before too many tombstones were found.

好吧，问起来比较简单；超过tombstone_failure_threshold会发生什么?

Well, to ask it simpler; what happens when the tombstone_failure_threshold is exceeded?

INFO [HintedHandoff:36] 2014-02-12 17:44:22,355 HintedHandOffManager.java (line 323) Started hinted handoff for host: fb04ad4c-xxxx-4516-8569-xxxxxxxxx with IP: /XX.XX.XXX.XX
ERROR [HintedHandoff:36] 2014-02-12 17:44:22,667 SliceQueryFilter.java (line 200) Scanned over 100000 tombstones; query aborted (see tombstone_fail_threshold)
ERROR [HintedHandoff:36] 2014-02-12 17:44:22,668 CassandraDaemon.java (line 187) Exception in thread Thread[HintedHandoff:36,1,main]
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
    at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
    at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
    at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
    at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
    at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
    at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
    at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1516)
    at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1335)
    at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
    at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
    at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
    at org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

忘记提及；运行Cassandra版本2.0.4

Forgot to mention; running Cassandra version 2.0.4

推荐答案

向Cassandra发出返回行(或列)范围的查询时，它必须扫描表以收集结果集(这称为一片).现在，已删除的数据以与常规数据相同的方式存储，除了在删除之前将其标记为逻辑删除.但是，表格读取器仍然必须对其进行扫描.因此，如果您周围有大量的墓碑，那么您将需要做大量的工作来满足您看似有限的切片.

When a query that returns a range of rows (or columns) is issued to Cassandra, it has to scan the table to collect the result set (this is called a slice). Now, deleted data is stored in the same manner as regular data, except that it's marked as tombstoned until compacted away. But the table reader has to scan through it nevertheless. So if you have tons of tombstones lying around, you will have an arbitrarily large amount of work to do to satisfy your ostensibly limited slice.

一个具体示例:假设您有两行带有聚类键1和3的行，以及十万个带有聚簇键2的无效行，它们位于表的第1行和第3行之间.现在，当您发出SELECT查询时，键应为> = 1和<. 3，您必须扫描100002行，而不是预期的两行.

A concrete example: let's say you have two rows with clustering keys 1 and 3, and a hundred thousand dead rows with clustering key 2 that are located in between rows 1 and 3 in the table. Now when you issue a SELECT query where the key is to be >= 1 and < 3, you'll have to scan 100002 rows, instead of the expected two.

更糟糕的是，Cassandra不仅要扫描这些行，还必须在准备响应时将它们存储在内存中.如果事情太遥远，这可能会导致节点上的内存不足错误，并且如果多个节点正在处理请求，它甚至可能导致多个故障，从而使整个集群瘫痪.为了防止这种情况的发生，如果服务检测到危险数量的逻辑删除，则该服务将中止查询.您可以随意进行此操作，但是如果在这些峰值期间您的Cassandra堆快要用完了，那就有风险了.

To make it worse, Cassandra doesn't just scan through these rows, but also has to accumulate them in memory while it prepares the response. This can cause an out-of-memory error on the node if things go too far out, and if multiple nodes are servicing the request, it may even cause a multiple failure bringing down the whole cluster. To prevent this from happening, the service aborts the query if it detects a dangerous number of tombstones. You're free to crank this up, but it's risky, if your Cassandra heap is close to running out during these spikes.

此异常是在最近的修复程序中引入的，该修复程序在2.0.2中首次可用. 此处是错误条目，描述了更改要解决的问题.以前，一切都会好起来，直到您的一个节点或可能几个节点突然崩溃.

This exception was introduced in a recent fix, first available in 2.0.2. Here is the bug entry describing the problem the change was trying to address. Previously everything would have been just fine, until one of your nodes, or potentially several, suddenly crashed.

如果只是某些选择被中止，那没什么大不了的. 但这是假设中止意味着上限"，因为查询会停止 过早地返回它设法收集到的所有实时数据 发现了太多的墓碑.

If it's merely some selects being aborted, it's not that big a deal. But that's assuming abort means 'capped' in that the query stops prematurely and returns whatever live data it managed to gather before too many tombstones were found.

查询没有返回有限的集合，它实际上完全删除了请求.如果您想减轻压力，也许值得以宽限期相同的节奏进行批量行删除，因此您不必每周都有大量的墓碑涌入.

The query doesn't return a limited set, it actually drops the request completely. If you'd like to mitigate, maybe it's worth doing your bulk row deletion at the same cadence as the grace period, so you don't have this huge influx of tombstones every week.

这篇关于达到墓碑限制时会发生什么的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！