提交日志不断增长中的问题 [英] Issues in growing commit logs

问题描述

我有一台提交日志不断增加到7.8 GB并仍在增长的机器，我检查了一个属性 commitlog_total_space_in_mb：8192 ，该属性在cassandra.yaml中进行了注释。我怀疑它一定是默认值。

1）提交日志的大小增加了什么问题？

2）是否表示未达到内存不足的阈值？

已编辑：

memtable_cleanup_threshold = 1 /（memtable_flush_writers + 1）*（memtable_offheap_space_in_mb + memtable_heap_space_in_mb）

其中推荐值为

  memtable_flush_writers-磁盘数或磁盘数较小最小为2且最大为8的核心，因此在我们的情况下为'8'
 
 memtable_offheap_space_in_mb-堆大小的1/4，因此在我们的情况下为2GB 
 memtable_heap_space_in_mb-堆大小的1/4，因此在我们的示例中为2GB 
  

是，

  memtable_cleanup_threshold = 1 /（8 + 1）* 4096 
 memtable_cleanup_threshold = 455MB 
  

为什么没有刷新到455 MB并删除提交日志？

I have a machine the commit logs keeps increasing upto 7.8 GB and still growing, I checked a property commitlog_total_space_in_mb: 8192 which is commented in cassandra.yaml. I suspect it has to be the default.

1) What is the problem in growing commit log size?
2) Is it says my memtable treshold is not reached?

EDITED :

memtable_cleanup_threshold = 1/(memtable_flush_writers + 1) * (memtable_offheap_space_in_mb + memtable_heap_space_in_mb)

where recommended values are,

memtable_flush_writers -  Smaller of number of disks or number of cores with a minimum of 2 and a maximum of 8, so in our case it is '8'

memtable_offheap_space_in_mb - 1/4 of the heap size, so in our case it is 2GB
memtable_heap_space_in_mb - 1/4 of the heap size, so in our case it is 2GB

so calculation will be,

 memtable_cleanup_threshold = 1/(8 + 1) * 4096
 memtable_cleanup_threshold = 455MB

Why does it didn't flush reaching to 455 MB and remove the commit log?

解决方案

Yes, 8192MB (or 1/4 of log file disk space, whichever is smaller -- could apply if you have a smaller server) is the default. Source: Cassandra documentation on commitlog_total_space_in_mb.

To answer your questions:

(1) If the commitlog files continue to grow, you can run out of disk space.

(2) The configured threshold has not yet been met.

Edited after your additional questions to add:

Commitlogs aren't deleted when memtables are flushed.

Note the file size is preallocated based on your configuration size -- I think you already figured that out, but noting that here if anyone else tries to observe the file size via ls or similar.

If you nodetool drain or restart, they will be cleared. Otherwise, they will continue to grow to the max size and rotate around.

Here is a test to see what happens if you force a flush:

nodetool tablestats keyspace.table | grep "Memtable data size"
Memtable data size: 1292049

cat /var/lib/cassandra/commitlog/CommitLog-A.log | wc -l
10418

cat /var/lib/cassandra/commitlog/CommitLog-B.log | wc -l
0

nodetool flush

nodetool tablestats keyspace.table | grep "Memtable data size"
Memtable data size: 0

cat /var/lib/cassandra/commitlog/CommitLog-A.log | wc -l
10419

cat /var/lib/cassandra/commitlog/CommitLog-B.log | wc -l
0

nodetool drain

nodetool tablestats keyspace.table | grep "Memtable data size"
Memtable data size: 0

cat /var/lib/cassandra/commitlog/CommitLog-A.log | wc -l
no such file

cat /var/lib/cassandra/commitlog/CommitLog-B.log | wc -l
0

You see similar results if it flushes automatically based on memtable configuration. Throughout the following observed flushes, the commitlog was not purged either:

这篇关于提交日志不断增长中的问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！