Cassandra 在启动后几分钟内因内存不足而崩溃 [英] Cassandra crashes with Out Of Memory within minutes after starting

问题描述

我们在 AWS 上使用 EC2Snitch 有一个具有 3 个节点和复制因子 3 的 Cassandra 集群.

We have a Cassandra cluster with 3 nodes and replication factor 3 on AWS using EC2Snitch.

实例类型为 c5.2xlarge(8 核和 16GB RAM).

Instance type is c5.2xlarge (8 core and 16GB RAM).

集群一直运行良好，但突然从昨天晚上开始，所有节点上的 cassandra 进程开始崩溃.它们被设置为自动重启，但随后在启动后 1 或 2 或 3 分钟内因内存堆空间不足错误而崩溃.

The cluster had been working fine but suddenly since yesterday evening, the cassandra process on all the nodes started crashing. They are set to restart automatically but then they crash with Out of Memory Heap Space error in 1 or 2 or 3 minutes after start.

堆配置:

MAX_HEAP_SIZE="4G"
HEAP_NEWSIZE="800M"

此后，我们尝试将节点大小增加到 r5.4x 或 128 GB 内存并分配 64 GB 堆，但仍然发生同样的事情，无论 3 个节点都启动还是一次仅启动一个节点.我们可以注意到，第一次垃圾收集在一段时间后发生，然后在几秒钟内连续发生，无法释放更多内存并最终崩溃.

After this, we tried increasing the node size to r5.4x or 128 GB memory and assigned 64GB Heap but still the same thing happens, irrespective of all 3 nodes being started or only one node being started at a time. We could note that first garbage collection happens after some time and then consecutively within seconds, failing to free any further memory and eventually crashing.

我们不确定启动后立即将哪些内容拉入内存.

We are not sure what is being pulled to memory immediately after starting.

其他参数:

• Cassandra 版本:2.2.13
• 数据库大小为 250GB
• hinted_handoff_enabled:真
• commitlog_segment_size_in_mb: 64
• memtable_allocation_type:offheap_buffers

这里的任何帮助，将不胜感激.

Any help here, would be appreciated.

查询时发现有特定表，导致casssandra节点崩溃.

We found that there is particular table when queried, it causes the casssandra node to crash.

cqlsh:my_keyspace> select count(*) from my_table ;
ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

所以我们认为，这与此特定表中的数据损坏/庞大有关.谢谢.

So we think, it is related to the data being corrupt/huge in this particular table. Thanks.

推荐答案

一些快速观察:

• 如果您要构建新集群，请使用最新的 3.11.x 版本.在 2.2 上构建新的没有意义.
• 根据您的设置，您似乎正在使用 CMS GC.如果您不太熟悉 GC 调整，您可能会通过切换到 G1 获得更高的稳定性，并且不要指定 HEAP_NEWSIZE(G1 自己计算 Eden 大小).
• 如果您坚持使用 CMS，那么将 HEAP_NEWSIZE 设置为 100mb x 内核的指南是错误的.为避免新旧代升级，请将 HEAP_NEWSIZE 设置为总堆大小的 40%-50%，并将 MaxTenuringThreshold 增加到 6-8 之类的值.
• 在带有 CMS GC 的 16GB RAM 机器上，我将使用 8GB 堆，并将 memtable_allocation_type: offheap_buffers 翻转回 heap_buffers.
• 将 commitlog_segment_size_in_mb 设置回 32.通常当人们需要处理它时，它会降低它，除非您还更改了 max_mutation_size_in_kb.
• 您没有提到崩溃发生时应用程序正在做什么.我怀疑正在发生写入繁重的负载.在这种情况下，您可能需要 3 个以上的节点，或者在应用程序端查看速率限制进行中的写入次数.

• If you're building a new cluster, use the latest 3.11.x version. There's no point in building new on 2.2.
• Based on your settings, it looks like you're using CMS GC. If you're not overly familiar with GC tuning, you may get more stability by switching to G1, and not specifying a HEAP_NEWSIZE (G1 figures out Eden sizing on its own).
• If you're stuck on CMS, the guidance for setting HEAP_NEWSIZE at 100mb x cores, is wrong. To avoid new->old gen promotion, set HEAP_NEWSIZE at 40%-50% of total heap size and increase MaxTenuringThreshold to something like 6-8.
• On a 16GB RAM machine with CMS GC, I would use a 8GB heap, and flip memtable_allocation_type: offheap_buffers back to heap_buffers.
• Set commitlog_segment_size_in_mb back to 32. Usually when folks need to mess with that, it's to lower it, unless you've also changed max_mutation_size_in_kb.
• You haven't mentioned what the application is doing when the crash happens. I suspect that a write-heavy load is happening. In that case, you may need more than 3 nodes, or look at rate-limiting the number of in-flight writes on the application side.

可帮助您的其他信息:

CASSANDRA-8150 - 关于良好 JVM 设置的 Cassandra 提交者讨论.

CASSANDRA-8150 - A Cassandra committer discussion on good JVM settings.

Amy 的 Cassandra 2.1 调整指南 - AmyTobey 的管理指南对集群配置的良好默认设置有很多智慧.

Amy's Cassandra 2.1 Tuning Guide - Amy Tobey's admin guide has a lot of wisdom on good default settings for cluster configuration.

编辑

我们使用的是 G1 GC.

We are using G1 GC.

不要使用 G1 设置堆新大小 (Xmn) 非常非常重要.确保被注释掉.

It is very, VERY important that you not set a heap new size (Xmn) with G1. Make sure that gets commented out.

从 my_table 中选择 count(*) ;

select count(*) from my_table ;

是的，未绑定的查询(没有 WHERE 子句`的查询)绝对会给节点带来不必要的压力.特别是如果桌子很大.这些类型的查询是 Cassandra 不擅长的.找到使用/需要这个结果的方法.

Yes, unbound queries (queries without WHERE clauses`) will absolutely put undue stress on a node. Especially if the table is huge. These types of queries are something that Cassandra just doesn't do well. Find a way around using/needing this result.

您可能可以通过将分页大小设置得更小(驱动程序端)或使用 Spark 之类的东西来设计此功能.或者可能通过令牌范围查询，并在应用程序端汇总结果.但你不这样做会好得多.

You might be able to engineer this to work by setting your paging size smaller (driver side), or by using something like Spark. Or maybe with querying by token range, and totaling the result on the app-side. But you'll be much better off not doing it.

这篇关于Cassandra 在启动后几分钟内因内存不足而崩溃的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！